:W&Bᾪ•?wM nmftypalª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.02yù•ù‚" 0wzo5rpb810q nmftypalŠÕ nmftypaljuju11-uc-san-diegoSPAtop"˜ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.2708857790552682 epochs‚10  hidden_dim‚64 ' learning_rate‚0.0002230140447417664  num_attention_heads‚4  num_branch_encoder_layers‚7  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_vanilla_nmftypalb88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-2-spfhcŠ ·ûâÔð×Öºš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64Ri´ï7 nmftypal&  _runtime‚0  _wandb.runtime‚0èUÑijÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ¶ûâÔ蘮Ê*-of*=/data/spatop/options_files/sweep_tttt_jj/config_nmftypal.json*-l* /scratch/logs*-n*jjtttt_vanilla_nmftypal2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-2-spfhcr/opt/conda/bin/pythonº bq294cp21lmj5zpt7s78yxef5hw1ymjyJþñÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ •ì§!ª€àäÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-1512559d-7724-700d-39a3-dab660c9a52eÊ13.3º bq294cp21lmj5zpt7s78yxef5hw1ymjyµª½Ç2  requirements.txt ,á«W nmftypalZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64Çb2SY nmftypalZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64ßs6Ï*  nmftypalb trainer/global_step2É5Ȉ\  nmftypalZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64§Km¬/  nmftypalb*"trainer/global_step2e·K  nmftypalj: ¸ûâÔ¸Ú‰*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ‹5Á´“  nmftypalj  ¸ûâÔˆÏì”î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 498 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 13.6 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ sySÇ\ nmftypaljÊ ¸ûâÔÀïæ•·Trainable params: 14.3 M Non-trainable params: 86 Total params: 14.3 M Total estimated model params size (MB): 57.158 Modules in train mode: 5901 Modules in eval mode: 0 Total FLOPs: 0 ûÂxµq nmftypalj` ¸ûâÔ öƒ–Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ¾®¥{ nmftypaljj ¸ûâÔò”–Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). áñ;y nmftypaljh ¸ûâԨϖXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ¹9e€ nmftypaljo ¸ûâÔèò¦–_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). Ùd%Õz nmftypalji ¸ûâÔØŒ®–Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). Þ¿R˜t nmftypaljc ¸ûâԈⶖSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). ³ýßq nmftypalj` ¸ûâԈǿ–Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ó"t nmftypaljc ¸ûâÔÐ…Í–Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). Ún nmftypalj] ¸ûâÔ€áÓ–Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ^ñöwm nmftypalj\ ¸ûâÔà¦Ü–Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). PÛ\‰_ nmftypal*Í  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"úŽÈ¿ë nmftypaljÙ ¸ûâÔøüþœÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ò?õmN nmftypalj¼ ¸ûâÔ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ]ÿƒ?ë nmftypaljÙ »ûâÔàÒ´È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. vCÌÛP nmftypalj¾ »ûâÔ¸Ö®´­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. 6Ò÷ׂ(:† ÇûâÔž¦þÒ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4892 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ÑyÞÒ‚(:È ÇûâÔÕ¢›Ý disk.sda1.out‚0 disk.rbd2.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.768932342529297 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.rbd1.out‚0 proc.cpu.threads‚17( disk./.usagePercent‚71.5333494973077% disk./.usageGB‚1065.8986015319824 disk.dm-0.out‚ 339.2578125 disk.rbd2.in‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2747.421875 network.sent‚33436 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 disk.dm-1.out‚ 11.5078125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 11.5078125" disk.nvme0c0n1.out‚ 339.2578125$ memory_percent‚93.79332065582275 disk.sda.out‚ 11.5078125 disk.sda3.in‚0 disk.rbd0.out‚0 disk.nvme0n1.out‚ 339.2578125 disk.dm-0.in‚0) proc.memory.availableMB‚ 1016.90234375 cpu‚24.967660608145795 network.recv‚42515 disk.sda2.in‚0 disk.sda2.out‚0ª¤ßØ ‚(:† ÖûâÔŽÉìÒ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.328' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8656 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µàîå!‚(:Û ÖûâÔ»ÞŒÕ disk.dm-0.out‚ 353.046875 disk.sda3.out‚ 12.19140625 network.sent‚55859 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.06298828125% disk./.usageGB‚1065.8926124572754 disk.nvme0n1.in‚0 disk.dm-0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 12.19140625$ memory_percent‚93.81065368652344 disk.rbd1.in‚0% proc.memory.availableMB‚ 1014.0625) disk./.usagePercent‚71.53294756547895 disk.sda.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 cpu‚24.992564175277277 network.recv‚69313! disk.nvme0c0n1.out‚ 353.046875# proc.memory.rssMB‚ 2747.44921875 disk.nvme0n1.out‚ 353.046875 disk.rbd2.out‚ 1.6796875 disk.rbd3.in‚0* proc.memory.percent‚16.769099235534668 disk.nvme0c0n1.in‚0 disk.sda.out‚ 12.19140625 disk.rbd0.in‚0¿Tq"‚(:† åûâÔɬŸÓ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3596 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·2©qå#‚(:Û åûâÔȽ”Ö% disk./.usageGB‚1065.8928565979004 disk.sda1.in‚0 disk.sda3.out‚14.90625 disk.rbd1.out‚ 0.06298828125 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.out‚14.90625) proc.memory.availableMB‚ 1013.92578125 network.sent‚65547 disk.sda2.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2747.4609375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚0 disk.dm-0.out‚ 451.431640625$ disk.nvme0c0n1.out‚ 451.431640625 disk.rbd2.out‚ 1.6796875 disk.rbd3.in‚0 disk.dm-1.out‚14.90625 disk.dm-0.in‚0# memory_percent‚93.8114881515503 cpu‚25.003846063031574 disk.drbd1777.in‚0 disk.rbd3.out‚0" disk.nvme0n1.out‚ 451.431640625 network.recv‚81201) disk./.usagePercent‚71.53296394996114 disk.drbd1777.out‚0 disk.sda3.in‚0( proc.memory.percent‚16.76917076110843c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JPªèƒŠ-&) _wandbruntime‚51  _runtime‚51ˆÃOÀ'» & lr-AdamW/pg1‚9.038617197970682e-06 & lr-AdamW/pg2‚9.038617197970682e-06 trainer/global_step‚49 % _timestamp‚1.7883949874534013e+09 _runtime‚ 51.013293248 _step‚0›¹5VÂ(½ % _timestamp‚1.7883949874534013e+09 _runtime‚ 51.013293248 _step‚0 & lr-AdamW/pg1‚9.038617197970682e-06 & lr-AdamW/pg2‚9.038617197970682e-06 trainer/global_step‚49JÅm i)be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPî¨Äs*bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP;,Xh+bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP7G?$g,bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP—Üh4f-bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP˜i£e.ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPÂeë7k/bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP=!¢s0bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÛ¤Vq1bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPÍ°rÏn2bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPBP\g3bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP΄ [n4bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP¢»·l5bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPüä®p6bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPïÀk7bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP„ÐCîr8bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP M2ƒm9bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP<ç†.n:bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPJ)Œv;br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP4Oz)fb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPCí%¥l?bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPWÖÉùl@bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP«ð!nAbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP:sugEbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP8Dd,nFbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP¬ˆÛrGbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP#ÖVMaHb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPß„nhIbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPçÃRnJbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP«hôëjKbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP5ÕÙŽmLbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPô ¹ sMbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP~Wñ`Nb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP¢Çž`nObj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPÏ @FnPbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPoÂefQbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP ¬oRbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP"@p²hSbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPRäyjTbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP6kpUbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP·“¿mVbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPuÏ;ÜjWbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPb°m¶iXbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP NXkYbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPÑŒY‡lZbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPhT}xq[bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPýÓÏo\bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP€Óƒ±a]b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPex*{j^bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP¨X ‰q_bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPðn|n`bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP‰ìmRhabd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPö‹8dbb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPíÅ9qccb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPÆ -Niobe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP zýqepba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPS]¦/hqbd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP÷ã‘grbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP¬]Änsbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPHY‡jtbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPû¬á€rubn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPøsV-kvbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP7Öøôowbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPUg 0pxbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPrjclybh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPg§0Óezba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPÔ˜W{bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP[<èt|bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPFÒýÐk}bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP‰ºb~b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPÜÀEfbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP8³¿y€bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPRš7úab\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP6rtWk‚bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP Ýrhƒbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP7Õׂf„ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP‚‚j…be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP»Cdt†bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP§²Í—n‡bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPXßN‹qˆbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP/6J`‰b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP” pŠbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPèªó×l‹bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPÆv5qŒbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPÜ=d ybt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPûx[uŽbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPþ~«rbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPù¶MJrbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPº Zq‘bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPª Oæi’bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP?© j“be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP§ë‰—t”bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÀû«®m•bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPJ°æh–bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP9#¨m—bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPQ[?Òc˜b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPôà Ht™bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP¯â£2ešb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPÄIà_›bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP¶sc©oœbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP„=³/lbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP ö¥jžbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPúvÅ!gŸbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPã3*g bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPü”öj¡be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP×öÛs¢bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP¥à k£bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPˆ8.¤) _wandbruntime‚51  _runtime‚51’'!µÂ¥€O HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 _runtime‚ 51.014481186 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1965838065152 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1965838065152 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2241296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883949874543347e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1295199 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚544072 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1762159582208 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1876559531520 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 89296536576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2182188618752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30533 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1349 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2182212558848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30308 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 89296536576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521396224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 90920081920 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 90655875584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1966088886784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268397568 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1851431876558 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚887966 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1876559531520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30533 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚724 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚921411 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1295199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1319883 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1762176621568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30308 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚888690 \LDeviceStatsMonitor.on_train_batch_ó2ä›Ästart/allocated_bytes.small_pool.allocated‚ 89529355264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚544070 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346885 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1319883 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264206336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2216610 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1876541528576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚343896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚49 _step‚1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873437696 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚921413 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚344618 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1851185874894 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497293 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 89255254990 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521396224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288146432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚921413 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 89529355264 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2272844494336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1966088886784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1876541528576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚921411 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2273132640768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873437696 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30308 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚888690 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 89529355264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚544070 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346885 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1319883 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264206336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2216610 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1876541528576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚343896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚49 _step‚1 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873437696 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚921413 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚344618 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1851185874894 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497293 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 89255254990 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521396224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 288146432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚921413 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 89529355264 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2272844494336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1966088886784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1876541528576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚921411 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2273132640768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873437696 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 _runtime‚ 51.014481186 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1965838065152 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1965838065152 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2241296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883949874543347e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1295199 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚544072 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1762159582208 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1876559531520 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 89296536576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 2182188618752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30533 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1349 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2182212558848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30308 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 89296536576>fÑ…b§b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP–')m¨bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP¦=Rj©be BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP”Úîhªbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPQewA`«b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPR:r/k¬bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPb(w­br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP‹75cg®bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPk¸PÃe¯b` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP»„¸^°bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP%náa±b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP ”ü]²bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP‘á—Éb³b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP°Æ‰kg´bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPbÍí rµbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP™t6p¶bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPú~Ïïm·bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPç&¢=m¸bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPuÒO¬_¹bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP;Kæfºba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPHß´i»bd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPgr’Še¼b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP™¶âo½bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP¶³{ip¾bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP¡¼çq¿bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPÝ©{ÖoÃbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPB¸¿hÄbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPšÎmŸuÅbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP‡s3biÆbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP?žÆrÇbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPôÖpÈbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP¯hupÉbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPìfçîkÊbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPN«KcËb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JP“oWÕfÌba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÓJ’]kÍbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPÔÕgrÎbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPgžDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP?öAf…ba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPõô÷Ef†ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP•ìg‡bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP‚âmˆbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP@%ò¯h‰bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPžžŠlŠbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPÙánül‹bg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPá•F?hŒbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPMÆyqbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPŸÓó¼fŽba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP ¾…‚eb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP1%/€fba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPØÎ.aV‘bQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP$ò¯že’b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP’™ °j“be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPP\Sm”bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPOéåço•bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP–¹¬Zr–bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPô‘]k—bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP ˜\ph˜bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP°±õ)n™bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPOÒ‡rjšbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPПÑo›bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP¼Sx±lœbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP®¨kbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP>ßxäwžbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP SšrŸbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPMù~'m bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPË ëÉr¡bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP¨ý9.¢) _wandbruntime‚51  _runtime‚51)åªT£‰M F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2005040338432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1913683818496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1797086275584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2285576 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2318051847680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚49 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚905316 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚939590 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1349 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1913701821440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1913701821440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _runtime‚ 51.925139144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2225591468544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚939592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268397568 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2285576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1913683818496 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 92742220800 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1887916301710 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2260864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1797103314944 UHDeviceSt&ü(4;atsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚350593 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚939592 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1321274 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚351266 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 91058988638 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2318333689344 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1888162303582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788394988364638e+09 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2005040338432 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 91338516992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚554723 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚939590 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2225567528448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1345984 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1321 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 92484319232 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 92484319232 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2005040338432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1913683818496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1797086275584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2285576 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2318051847680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚49 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚905316 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚939590 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1349 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1913701821440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1913701821440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _runtime‚ 51.925139144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2225591468544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚939592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268397568 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2285576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1913683818496 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 92742220800 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1887916301710 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2260864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1797103314944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚350593 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚939592 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1321274 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚351266 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 91058988638 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2318333689344 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1888162303582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788394988364638e+09 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2005040338432 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 91338516992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚554723 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚939590 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2225567528448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1345984 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1321 E loss/SRqqt2/assignment_loss"trainer/global_step2:JP¾Õ@©b; loss/FRt1/detection_loss"trainer/global_step2:JPí~3Bªb= loss/SRqqt2/detection_loss"trainer/global_step2:JP™öçC«b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP&†èC¬b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP ‡zA­b< loss/FRt2/assignment_loss"trainer/global_step2:JP4“ñMA®b< loss/FBt1/assignment_loss"trainer/global_step2:JPòž(@¯b; loss/FRt3/detection_loss"trainer/global_step2:JPxoû¶@°b; loss/FBt3/detection_loss"trainer/global_step2:JP¾Æú7±b2 loss/total_loss"trainer/global_step2:JPÂŽtwA²b< loss/FBt2/assignment_loss"trainer/global_step2:JPôþ‚Ý@³b; loss/FRt2/detection_loss"trainer/global_step2:JPHór‰B´b= loss/SRqqt4/detection_loss"trainer/global_step2:JPl±EAµb< loss/FRt1/assignment_loss"trainer/global_step2:JP1ôxˆA¶b< loss/FRt3/assignment_loss"trainer/global_step2:JP–Å…C·b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPv³Ñ”A¸b< loss/FBt3/assignment_loss"trainer/global_step2:JP‡™Þ@¹b; loss/FRt4/detection_loss"trainer/global_step2:JP´ 2>Bºb= loss/SRqqt1/detection_loss"trainer/global_step2:JP5{B»b= loss/SRqqt3/detection_loss"trainer/global_step2:JPô˜—l-¼b( epoch"trainer/global_step2:JP¥Û¤@½b; loss/FBt2/detection_loss"trainer/global_step2:JPµè£@¾b; loss/FBt1/detection_loss"trainer/global_step2:JPRÙŠ\.¿) _wandbruntime‚51  _runtime‚51 ÒFkHÀ /loss/FRt1/assignment_loss‚7.402824078287397 /loss/FRt3/assignment_loss‚7.305799529665992 1loss/SRqqt4/assignment_loss‚5.413370166506086 0loss/FBt3/assignment_loss‚2.7802351760864257 .loss/FRt4/detection_loss‚4.371233631403018 1loss/SRqqt1/detection_loss‚2.5195192859479874 0loss/SRqqt3/detection_loss‚3.090280819210139 epoch‚0 $ _timestamp‚1.788394988366677e+09 .loss/FBt2/detection_loss‚5.368399876005509 .loss/FBt1/detection_loss‚8.345622929930688 0loss/FBt4/assignment_loss‚2.7606959342956543 .loss/FBt4/detection_loss‚4.920956058537259 _step‚3 .loss/FRt4/assignment_loss‚7.33756257326175 1loss/SRqqt2/assignment_loss‚5.441816441556241 /loss/FRt1/detection_loss‚3.1127885465643237 1loss/SRqqt2/detection_loss‚3.5572343506711595 trainer/global_step‚49 1loss/SRqqt1/assignment_loss‚5.480657194599961 1loss/SRqqt3/assignment_loss‚5.382194154912775 /loss/FRt2/assignment_loss‚7.305920649547966 /loss/FBt1/assignment_loss‚2.774679625034332 /loss/FRt3/detection_loss‚3.9715853762768565 .loss/FBt3/detection_loss‚7.221850376129151 &loss/total_loss‚115.99007187470151 0loss/FBt2/assignment_loss‚2.7439259571187637 /loss/FRt2/detection_loss‚3.5142361418325074 0loss/SRqqt4/detection_loss‚3.866683001319567 _runtime‚ 51.926678945lÙóLÁÆ trainer/global_step‚49 1loss/SRqqt1/assignment_loss‚5.480657194599961 1loss/SRqqt3/assignment_loss‚5.382194154912775 /loss/FRt2/assignment_loss‚7.305920649547966 /loss/FBt1/assignment_loss‚2.774679625034332 /loss/FRt3/detection_loss‚3.9715853762768565 .loss/FBt3/detection_loss‚7.221850376129151 &loss/total_loss‚115.99007187470151 0loss/FBt2/assignment_loss‚2.7439259571187637 /loss/FRt2/detection_loss‚3.5142361418325074 0loss/SRqqt4/detection_loss‚3.866683001319567 _runtime‚ 51.926678945 /loss/FRt1/assignment_loss‚7.402824078287397 /loss/FRt3/assignment_loss‚7.305799529665992 1loss/SRqqt4/assignment_loss‚5.413370166506086 0loss/FBt3/assignment_loss‚2.7802351760864257 .loss/FRt4/detection_loss‚4.371233631403018 1loss/SRqqt1/detection_loss‚2.5195192859479874 0loss/SRqqt3/detection_loss‚3.090280819210139 epoch‚0 $ _timestamp‚1.788394988366677e+09 .loss/FBt2/detection_loss‚5.368399876005509 .loss/FBt1/detection_loss‚8.345622929930688 0loss/FBt4/assignment_loss‚2.7606959342956543 .loss/FBt4/detection_loss‚4.920956058537259 _step‚3 .loss/FRt4/assignment_loss‚7.33756257326175 1loss/SRqqt2/assignment_loss‚5.441816441556241 /loss/FRt1/detection_loss‚3.1127885465643237 1loss/SRqqt2/detection_loss‚3.5572343506711595Yy‘ʑ‚(:† ôûâÔóÜðÒ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.086' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8172 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã,/öÂ(:ë ôûâÔ‚õŸÕ disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚0 network.sent‚93017 network.recv‚115450 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 26.94140625& proc.memory.availableMB‚ 1011.46875# proc.memory.rssMB‚ 2747.49609375 disk.drbd1777.in‚0 disk.rbd0.in‚0) proc.memory.percent‚16.76938533782959 disk.dm-0.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚550.52978515625 proc.cpu.threads‚17 disk.sda.out‚ 26.94140625 disk.rbd3.out‚0 disk.rbd1.in‚0% disk./.usageGB‚1065.8925895690918 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚71.53294602943375! disk.dm-0.out‚550.52978515625 disk.rbd1.out‚ 0.06298828125 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚550.52978515625 disk.rbd2.out‚ 3.35546875 disk.sda2.in‚0 disk.sda3.out‚ 26.94140625$ memory_percent‚93.82648468017578 cpu‚25.00948827355239»8 /‘Ä‚(:† ƒüâÔ¥›‚Ó gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.434' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0868 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ëõîÅ‚(:ã ƒüâÔÝܪ× disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚0 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.06298828125 disk.nvme0c0n1.in‚0! disk.dm-0.out‚551.55908203125 cpu‚25.01686658581212 disk.sda3.out‚41.65625 network.recv‚121044) disk./.usagePercent‚71.53294961353922 disk.nvme0n1.in‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚551.55908203125) proc.memory.percent‚16.76943302154541 disk.rbd2.out‚ 4.8671875 disk.dm-1.out‚41.65625 disk.sda1.out‚0( proc.memory.availableMB‚ 1012.3828125% disk./.usageGB‚1065.8926429748535 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚551.55908203125 disk.sda.out‚41.65625# memory_percent‚93.8209056854248 proc.cpu.threads‚17 network.sent‚142718 disk.sda3.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2747.50390625™_R‘Æ‚(:† ’üâÔªãèÒ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.561' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1122 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àgÿö÷Ç‚(:ì ’üâÔŠ¤×Õ disk.nvme0n1.in‚0 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0) proc.memory.percent‚16.76945686340332 network.recv‚126624 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1012.48828125 network.sent‚150066$ disk.nvme0n1.out‚650.53759765625 disk.rbd3.out‚0 disk.dm-0.in‚0 disk.rbd2.out‚ 5.03515625 disk.dm-1.out‚ 42.5234375) disk./.usagePercent‚71.53295857380292 disk.sda.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 0.06298828125 cpu‚25.021865451267537 disk.sda.out‚ 42.5234375 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2747.5078125 disk.sda3.out‚ 42.5234375 disk.rbd1.in‚0& disk.nvme0c0n1.out‚650.53759765625 disk.rbd0.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1065.8927764892578! disk.dm-0.out‚650.53759765625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0$ memory_percent‚93.82026195526123ü›Nà.È) _wandbruntime‚98  _runtime‚98QIº^Ãɽ _runtime‚ 98.636793077 _step‚4 ' lr-AdamW/pg1‚1.8261695971410152e-05 ' lr-AdamW/pg2‚1.8261695971410152e-05 trainer/global_step‚99 % _timestamp‚1.7883950350769312e+09â ³ÇÊÁ ' lr-AdamW/pg1‚1.8261695971410152e-05 ' lr-AdamW/pg2‚1.8261695971410152e-05 trainer/global_step‚99 % _timestamp‚1.7883950350769312e+09 _runtime‚ 98.636793077 _step‚4iå…ß.Ë) _wandbruntime‚98  _runtime‚98 »:­VÌO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2626985 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3913675217408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4533389444096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3913675217408 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2626985 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3733674027520 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1763335 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 179226555894 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7883950350777135e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 180001189888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3733656024576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3913424395776 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 179768371200 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1076720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4351158043648 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 182509061120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3508494251008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1326 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3687966808566 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2602301 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3913424395776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1076722 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 trainer/global_step‚99 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1830363 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3733656024576 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 179768371200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 _step‚5 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1354 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2602301 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4457348 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4351134103552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4432662 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3733674027520 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚687345 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.s¿EÛQ@mall_pool.allocated‚252 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4533667104768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 253720576 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 182255340544 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 179455518198 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1830363 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1830361 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268397568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 180001189888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚686615¿”#š'Í”O OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2602301 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4457348 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4351134103552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4432662 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3733674027520 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚687345 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4533667104768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 253720576 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 182255340544 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 179455518198 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1830363 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1830361 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268397568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 180001189888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚686615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2626985 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3913675217408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4533389444096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3913675217408 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2626985 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3733674027520 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1763335 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 179226555894 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7883950350777135e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 180001189888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3733656024576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3913424395776 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 179768371200 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1076720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4351158043648 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 182509061120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3508494251008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1326 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3687966808566 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2602301 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 3913424395776 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1076722 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 trainer/global_step‚99 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1830363 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3733656024576 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 179768371200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 _step‚5 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1354 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¡à.Î) _wandbruntime‚99  _runtime‚99(f. &ÏšM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1326 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2628376 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4578595486720 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 184082473472 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1087373 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3952626669056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3770816317440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2628376 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 181810351616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 _runtime‚ 99.542231122 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1354 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1848540 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 trainer/global_step‚99 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 181810351616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4476916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1848542 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3952626669056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3724451233718 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3543420944384 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1781005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 181577519616 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4578881522688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1087375 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3770816317440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1848542 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3724697235590 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1848540 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚694330 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1781705 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 181577519616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4476916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3770798314496 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4501628 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 262095872 % _timestamp‚1.7883950359818034e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3543437983744 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 181259251846 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0øÓª·¤&ОM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 181810351616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4476916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1848542 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3952626669056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3724451233718 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3543420944384 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1781005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 181577519616 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4578881522688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1087375 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3770816317440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1848542 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3724697235590 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1848540 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚694330 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1781705 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 181577519616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4476916 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3770798314496 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4501628 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 262095872 % _timestamp‚1.7883950359818034e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3543437983744 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 181259251846 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1326 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2628376 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4578595486720 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 184082473472 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1087373 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3952626669056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3770816317440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2628376 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 181810351616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 _runtime‚ 99.542231122 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1354 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1848540 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 trainer/global_step‚99 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0.ǧþ.Ñ) _wandbruntime‚99  _runtime‚99ýQóUÒÀ .loss/FRt1/detection_loss‚3.400675139446846 .loss/FBt3/detection_loss‚4.178235282376408 &loss/total_loss‚116.79536129164791 /loss/FRt3/assignment_loss‚7.232970342636109 1loss/SRqqt3/assignment_loss‚5.356231441100438 0loss/FBt1/assignment_loss‚2.7335497912238624 /loss/FRt3/detection_loss‚3.2494854083657265 0loss/SRqqt2/detection_loss‚5.549795981744925 .loss/FRt1/assignment_loss‚7.30809293323093 2loss/SRqqt1/assignment_loss‚5.4860098203023275 /loss/FBt3/assignment_loss‚2.744119942188263 /loss/FBt4/detection_loss‚5.2633917117491364 _steÞÊ#ñp‚7 /loss/FRt2/assignment_loss‚7.209757037784742 1loss/SRqqt2/assignment_loss‚5.408647076288859 .loss/FRt2/detection_loss‚3.747370141679826 /loss/FRt4/assignment_loss‚7.282208430139642 1loss/SRqqt4/assignment_loss‚5.410529497507456 0loss/FBt4/assignment_loss‚2.7229972183704376 .loss/FBt2/detection_loss‚4.961172965594701 % _timestamp‚1.7883950359837527e+09 0loss/SRqqt4/detection_loss‚4.319583099033381 .loss/FBt1/detection_loss‚9.747273295241243 epoch‚0 _runtime‚ 99.543457342 /loss/FBt2/assignment_loss‚2.741624620982579 0loss/SRqqt3/detection_loss‚3.651440535361568 .loss/FRt4/detection_loss‚4.370035000145435 0loss/SRqqt1/detection_loss‚2.720164579153061 trainer/global_step‚99ž²³JÓÄ /loss/FRt3/assignment_loss‚7.232970342636109 1loss/SRqqt3/assignment_loss‚5.356231441100438 0loss/FBt1/assignment_loss‚2.7335497912238624 /loss/FRt3/detection_loss‚3.2494854083657265 0loss/SRqqt2/detection_loss‚5.549795981744925 .loss/FRt1/assignment_loss‚7.30809293323093 2loss/SRqqt1/assignment_loss‚5.4860098203023275 /loss/FBt3/assignment_loss‚2.744119942188263 /loss/FBt4/detection_loss‚5.2633917117491364 _step‚7 /loss/FRt2/assignment_loss‚7.209757037784742 1loss/SRqqt2/assignment_loss‚5.408647076288859 .loss/FRt2/detection_loss‚3.747370141679826 /loss/FRt4/assignment_loss‚7.282208430139642 1loss/SRqqt4/assignment_loss‚5.410529497507456 0loss/FBt4/assignment_loss‚2.7229972183704376 .loss/FBt2/detection_loss‚4.961172965594701 % _timestamp‚1.7883950359837527e+09 0loss/SRqqt4/detection_loss‚4.319583099033381 .loss/FBt1/detection_loss‚9.747273295241243 epoch‚0 _runtime‚ 99.543457342 /loss/FBt2/assignment_loss‚2.741624620982579 0loss/SRqqt3/detection_loss‚3.651440535361568 .loss/FRt4/detection_loss‚4.370035000145435 0loss/SRqqt1/detection_loss‚2.720164579153061 trainer/global_step‚99 .loss/FRt1/detection_loss‚3.400675139446846 .loss/FBt3/detection_loss‚4.178235282376408 &loss/total_loss‚116.79536129164791 œÔ‚(:‘ ¡üâÔÜÂóÒ gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.557' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.711399999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï,­×ôÕ‚(:é ¡üâÔòÈÏÕ network.recv‚130953 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚0$ memory_percent‚93.82288455963135! proc.memory.rssMB‚ 2747.515625 disk.sda.out‚ 57.3515625 disk.sda1.out‚0 disk.sda3.out‚ 57.3515625 disk.rbd1.out‚ 0.06298828125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚155966 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚753.84033203125 cpu‚25.023196968713652 proc.cpu.threads‚17) disk./.usagePercent‚71.53296574201387 disk.sda.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0% disk./.usageGB‚1065.8928833007812 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚753.84033203125 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 57.3515625) proc.memory.availableMB‚ 1012.05859375) proc.memory.percent‚16.76950454711914! disk.dm-0.out‚753.84033203125 disk.sda1.in‚0 disk.rbd2.out‚6.546875 disk.rbd3.in‚0}$Á:‘Ö‚(:† °üâÔŠÛ’Ó gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3096 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(àŠZ÷ׂ(:ì °üâÔ“°Õ disk.nvme0n1.in‚0) disk./.usagePercent‚71.53296190190086 disk.rbd1.in‚0 disk.rbd2.out‚ 6.7109375 disk.drbd1777.in‚0 disk.rbd3.out‚0 cpu‚25.024320545641963 disk.rbd3.in‚0 network.recv‚137247% disk./.usageGB‚1065.8928260803223 disk.dm-1.in‚0) proc.memory.availableMB‚ 1012.17578125# proc.memory.rssMB‚ 2747.51953125 disk.rbd2.in‚0$ memory_percent‚93.82216930389404 network.sent‚206105 disk.sda.out‚ 58.0234375 disk.sda3.in‚0 disk.rbd1.out‚ 0.06298828125 disk.rbd0.in‚0! disk.dm-0.out‚754.48388671875 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 58.0234375$ disk.nvme0n1.out‚754.48388671875 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚754.48388671875 disk.sda1.out‚0 disk.sda3.out‚ 58.0234375 disk.rbd0.out‚0 disk.dm-0.in‚0 disk.sda.in‚0) proc.memory.percent‚16.76952838897705‰ÿµ¸‘Ø‚(:† ¿üâÔù½„Ó gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2518 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â_[€ôÙ‚(:é ¿üâÔ§»ßÕ disk.sda2.out‚0 disk.rbd1.out‚ 0.06298828125 disk.dm-1.in‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 71.796875 cpu‚25.02658703891446 disk.sda.out‚ 71.796875 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚853.65185546875) proc.memory.availableMB‚ 1012.09765625 network.sent‚211710 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 71.796875" proc.memory.rssMB‚ 2747.5703125) disk./.usagePercent‚71.53296318193854 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0% disk./.usageGB‚1065.8928451538086 disk.drbd1777.out‚0 disk.rbd1.in‚0 network.recv‚141598 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0* proc.memory.percent‚16.769838333129883 proc.cpu.threads‚17! disk.dm-0.out‚853.65185546875 disk.drbd1777.in‚0 disk.rbd2.out‚ 8.28515625$ memory_percent‚93.82264614105225 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚853.65185546875 disk.sda1.out‚0 ãÜh0Ú+ _wandbruntime‚146  _runtime‚146æ‚2WÃÛ½ _step‚8 & lr-AdamW/pg1‚2.748477474484962e-05 & lr-AdamW/pg2‚2.748477474484962e-05 trainer/global_step‚149 % _timestamp‚1.7883950830284092e+09 _runtime‚ 146.588240267ª8«(ÇÜÁ % _timestamp‚1.7883950830284092e+09 _runtime‚ 146.588240267 _step‚8 & lr-AdamW/pg1‚2.748477474484962e-05 & lr-AdamW/pg2‚2.748477474484962e-05 trainer/global_step‚149sÕ­0Ý+ _wandbruntime‚146  _runtime‚146YH£ š'Þ”O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1327 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283952128 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 274198416384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5861010726400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 270240205824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 270240205824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6520079588352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268397568 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5524255738910 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3934087 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6794017992704 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2739313 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5861010726400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5590770520576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1033374 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2739313 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7883950830291739e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2642746 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1355 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5590788523520 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3909403 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5254845959168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5861261548032 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2642045 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3934087 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1032675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2739311 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1609372 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 260012032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5590770520576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304):qž'ߘO VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 270240205824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 270240205824 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6520079588352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268397568 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5524255738910 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3934087 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6794017992704 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2739313 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5861010726400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5590770520576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1033374 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2739313 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7883950830291739e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2642746 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1355 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5590788523520 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3909403 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5254845959168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5861261548032 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2642045 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3934087 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1032675 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2739311 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1609372 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 260012032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5590770520576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1327 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283952128 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 274198416384 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5861010726400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ˜é„0à+ _wandbruntime‚147  _runtime‚147ú !7fáŸM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 272049354240 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2757492 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1327 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6563482438144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2660251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5900212999680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5289755613184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6692968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1040226 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6692968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5627912810496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5627930813440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 276025866240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 trainer/global_step‚149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1620025 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 271459515054 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5899962164736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1039535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch ôPM?_end/active_bytes.all.peak‚ 6873438720 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6717680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6563458498048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3960188 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6839508304384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 272282186240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 271230552542 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3935478 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6717680 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2757490 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5900212999680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2659558 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268397568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 _step‚10 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5560986165726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3935478 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 272282186240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5289772652544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2757490 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1620023 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6717680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6563458498048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3960188 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6839508304384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 272282186240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 271230552542 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3935478 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6717680 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2757490 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5900212999680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2659558 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268397568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 _step‚10 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5560986165726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3935478 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 272282186240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5289772652544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2757490 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1620023 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 272049354240 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2757492 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1327 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6563482438144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2660251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5900212999680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5289755613184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6692968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1040226 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6692968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5627912810496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5627930813440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 276025866240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 trainer/global_step‚149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1620025 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 271459515054 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5899962164736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1039535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F0ì+ _wandbruntime‚194  _runtime‚194ü’ŸÄí¾ & lr-AdamW/pg1‚3.670785351828909e-05 & lr-AdamW/pg2‚3.670785351828909e-05 trainer/global_step‚199 % _timestamp‚1.7883951308678782e+09 _runtime‚ 194.427671642 _step‚122^‚mÈî & lr-AdamW/pg1‚3.670785351828909e-05 & lr-AdamW/pg2‚3.670785351828909e-05 trainer/global_step‚199 % _timestamp‚1.7883951308678782e+09 _runtime‚ 194.427671642 _step‚12 »'…0ï+ _wandbruntime‚194  _runtime‚194ºçûÒ›'ð•O JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7447885016576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7001163588608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2142020 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 365615308288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1355 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7361036672582 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 359856044614 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7808597057024 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5216505 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3522108 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 249526272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 360944859136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3648263 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7808847878656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1379390 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 273466368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8689025073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8689049013248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7447903019520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚13 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7808597057024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7447903019520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7447885016576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 % _timestamp‚1.7883951308686495e+09 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7360790670918 _runtime‚ 194.428640697 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 360712040448 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2142022 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 360712040448 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1355 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7361036672582 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 359856044614 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7808597057024 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5216505 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3522108 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 249526272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 360944859136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3648263 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7808847878656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1379390 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 273466368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8689025073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8689049013248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7447903019520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚13 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7808597057024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7447903019520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7447885016576 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 % _timestamp‚1.7883951308686495e+09 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7360790670918 _runtime‚ 194.428640697 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 360712040448 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2142022 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 360712040448 HDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3538998 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9099848242688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 362521188864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1355 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1386325 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7397767099606 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7847799330304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7847799330304 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7036107321344 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3666442 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7485027306496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 362754020864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7485027306496 % _timestamp‚1.7883951317866952e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8933732 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2152675 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2152673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8732403982848 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 367444259840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5267290 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8909020 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _step‚14 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8732427922944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 195.347022536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304ý®(|©&ö£M VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9099848242688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 362521188864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1355 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1386325 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7397767099606 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7847799330304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7847799330304 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7036107321344 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3666442 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7485027306496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 362754020864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7485027306496 % _timestamp‚1.7883951317866952e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8933732 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2152675 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2152673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8732403982848 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 367444259840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5267290 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8909020 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _step‚14 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8732427922944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 195.347022536 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5242580 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 361430815750 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 362521188864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8933732 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3538998Êiñù0÷+ _wandbruntime‚195  _runtime‚195é@R‡NøÈ 0loss/FBt2/assignment_loss‚2.2836263047324286 .loss/FBt1/detection_loss‚54.06162430842718 /loss/FBt3/detection_loss‚11.138690052554011 % _timestamp‚1.7883951317884507e+09 /loss/FRt1/assignment_loss‚7.217571871621268 _step‚15 0loss/FRt4/assignment_loss‚7.2820564183321865 /loss/FBt4/detection_loss‚1.8965492915354092 trainer/global_step‚199 1loss/SRqqt3/assignment_loss‚5.300756050061576 /loss/FBt1/assignment_loss‚2.562799612681071 1loss/SRqqt2/detection_loss‚3.7622617032638814 0loss/SRqqt4/detection_loss‚6.495930987596512 1loss/SRqqt4/assignment_loss‚5.435588340759278 /loss/FRt4/detection_loss‚5.1585975668647075 1loss/SRqqt3/detection_loss‚1.6339831637525106 /loss/FBt2/detection_loss‚10.210218894812796 epoch‚0 _runtime‚ 195.348146073 /loss/SRqqt1/detection_loss‚2.83949465507262 1loss/SRqqt2/assignment_loss‚5.426386079122854 0loss/FBt4/assignment_loss‚1.8692127157901894 /loss/FRt1/detection_loss‚3.6415010885435515 .loss/FRt2/detection_loss‚4.494561683974768 /loss/FRt3/detection_loss‚2.1253997659254256 %loss/total_loss‚167.3156957861964 /loss/SRqqt1/assignment_loss‚5.5008934073978 0loss/FBt3/assignment_loss‚2.5282968282699585 /loss/FRt2/assignment_loss‚7.363498160713597 0loss/FRt3/assignment_loss‚7.0861968343908135F´Z ùÌ epoch‚0 _runtime‚ 195.348146073 /loss/SRqqt1/detection_loss‚2.83949465507262 1loss/SRqqt2/assignment_loss‚5.426386079122854 0loss/FBt4/assignment_loss‚1.8692127157901894 /loss/FRt1/detection_loss‚3.6415010885435515 .loss/FRt2/detection_loss‚4.494561683974768 /loss/FRt3/detection_loss‚2.1253997659254256 %loss/total_loss‚167.3156957861964 /loss/SRqqt1/assignment_loss‚5.5008934073978 0loss/FBt3/assignment_loss‚2.5282968282699585 /loss/FRt2/assignment_loss‚7.3634981607äDž#E13597 0loss/FRt3/assignment_loss‚7.0861968343908135 0loss/FBt2/assignment_loss‚2.2836263047324286 .loss/FBt1/detection_loss‚54.06162430842718 /loss/FBt3/detection_loss‚11.138690052554011 % _timestamp‚1.7883951317884507e+09 /loss/FRt1/assignment_loss‚7.217571871621268 _step‚15 0loss/FRt4/assignment_loss‚7.2820564183321865 /loss/FBt4/detection_loss‚1.8965492915354092 trainer/global_step‚199 1loss/SRqqt3/assignment_loss‚5.300756050061576 /loss/FBt1/assignment_loss‚2.562799612681071 1loss/SRqqt2/detection_loss‚3.7622617032638814 0loss/SRqqt4/detection_loss‚6.495930987596512 1loss/SRqqt4/assignment_loss‚5.435588340759278 /loss/FRt4/detection_loss‚5.1585975668647075 1loss/SRqqt3/detection_loss‚1.6339831637525106 /loss/FBt2/detection_loss‚10.210218894812796™ƒøú‚(:… ŠýâÔ¬’´Ó gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚90.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0458 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ctZß1û‚(:¦ ŠýâÔæþ˜Õ network.recv‚166981 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.06640625 disk.dm-1.out‚ 131.50390625 disk.rbd0.in‚0 disk.dm-1.in‚ 0.06640625! disk.dm-0.out‚1155.0439453125$ proc.memory.availableMB‚1010.625* proc.memory.percent‚16.769886016845703 cpu‚25.03412925254418 network.sent‚318568 disk.rbd2.in‚0 disk.sda3.out‚ 131.50390625 disk.nvme0c0n1.in‚ 0.33203125 disk.sda.in‚ 0.06640625) disk./.usagePercent‚71.53318770054607 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 0.33203125 disk.sda.out‚ 131.50390625! proc.memory.rssMB‚ 2747.578125 proc.cpu.threads‚17$ disk.nvme0n1.out‚1155.0439453125% disk./.usageGB‚1065.8961906433105 disk.drbd1777.in‚0 disk.rbd2.out‚13.46142578125& disk.nvme0c0n1.out‚1155.0439453125 disk.nvme0n1.in‚ 0.33203125$ memory_percent‚93.83163452148438 disk.rbd3.out‚0 disk.rbd1.out‚ 0.06298828125 disk.rbd0.out‚0½ûü‚(:„ ™ýâÔÖ¨„Ó gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.03' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.006 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÙíä7ý‚(:¬ ™ýâÔ±×ÏÕ disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.06298828125' disk.nvme0c0n1.out‚1258.90283203125' proc.memory.availableMB‚ 1010.859375 disk.rbd1.in‚0 disk.rbd3.in‚0 network.sent‚357521 network.recv‚172509% disk./.usageGB‚1065.8964767456055 disk.nvme0n1.in‚ 0.33203125% disk.nvme0n1.out‚1258.90283203125 disk.sda.in‚ 0.06640625$ memory_percent‚93.83020401000977 disk.rbd0.in‚0" disk.dm-0.out‚1258.90283203125 disk.sda.out‚ 136.01171875! proc.memory.rssMB‚ 2747.578125 cpu‚25.03498724112691 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 136.01171875 disk.rbd2.out‚13.46142578125* proc.memory.percent‚16.769886016845703) disk./.usagePercent‚71.53320690111113 disk.sda2.out‚0 disk.dm-1.out‚ 136.01171875 disk.rbd0.out‚0 disk.dm-0.in‚ 0.33203125 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 0.33203125 disk.dm-1.in‚ 0.06640625 disk.sda1.out‚0Ø« ‘þ‚(:† ¨ýâÔ¬Ú—Ó gpu.0.gpu‚18.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1708 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä4¡Q9ÿ‚(:® ¨ýâÔŸÛÕ' proc.memory.availableMB‚ 1010.828125 network.recv‚177348 disk.rbd1.out‚ 0.06298828125 disk.dm-1.in‚ 0.06640625 disk.rbd0.out‚0 disk.drbd1777.in‚0 network.sent‚364299 disk.sda2.out‚0 disk.nvme0n1.in‚ 0.33203125 disk.dm-0.in‚ 0.33203125 disk.sda.out‚ 136.83203125 disk.sda1.out‚0# proc.memory.rssMB‚ 2747.58203125 disk.rbd3.out‚0' disk.nvme0c0n1.out‚1259.20361328125 disk.dm-1.out‚ 136.83203125$ memory_percent‚93.83039474487305) disk./.usagePercent‚71.53320357301318$ disk./.usageGB‚1065.896427154541 disk.rbd2.out‚15.21533203125 disk.sda2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.769909858703613 cpu‚25.035927951329665 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 0.33203125% disk.nvme0n1.out‚1259.20361328125 disk.rbd2.in‚0 disk.sda3.in‚ 0.06640625" disk.dm-0.out‚1259.20361328125 disk.sda3.out‚ 136.83203125 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625úÁ# 0€+ _wandbruntime‚242  _runtime‚242E9Ëáý & lr-AdamW/pg2‚4.593093229172856e-05 trainer/global_step‚249 $ _timestamp‚1.788395178977995e+09 _runtime‚ 242.537800709 _step‚16 & lr-AdamW/pg1‚4.593093229172856e-05@ÒEêÇ‚Á _runtime‚ 242.537800709 _step‚16 & lr-AdamW/pg1‚4.593093229172856e-05 & lr-AdamW/pg2‚4.593093229172856e-05 trainer/global_step‚249 $ _timestamp‚1.788395178977995e+09Uܨ±0ƒ+ _wandbruntime‚242  _runtime‚242¾À£M£'„O [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9305017515520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 451183875072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 457460738560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9756434209280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6548291 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6523607 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9197325602926 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2674672 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9197571604590 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9304999512576 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451416693760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4557213 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 457211212288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9304999512576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 249526272 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 _runtime‚ 242.538740999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6548291 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4557213 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2674670 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 273466368 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9756183387648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11105504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9756183387648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6523607 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1330 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10857970557952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 8747498257408 % _timestamp‚1.7883951789787307e+09 _step‚17 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 trainer/global_step‚249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11315455236608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 8747515296768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4395804 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10857994498048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 449827345518 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4557211z3¬§'…¡O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4557213 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 457211212288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9304999512576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 249526272 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 _runtime‚ 242.538740999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6548291 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4557213 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2674670 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 273466368 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9756183387648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11105504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9756183387648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6523607 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1330 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10857970557952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 8747498257408 % _timestamp‚1.7883951789787307e+09 _step‚17 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 trainer/global_step‚249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11315455236608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 8747515296768 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4395804 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10857994498048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 449827345518 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4557211 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9305017515520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 451183875072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 457460738560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9756434209280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6548291 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6523607 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9197325602926 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2674672 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9197571604590 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9304999512576 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451416693760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307+L¡0†+ _wandbruntime‚243  _runtime‚243”ò0#‡¦M >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2685323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283938816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4413175 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11360386188800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8782441990144 trainer/global_step‚249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 451631078958 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4413860 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11149784 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11360670127616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11149784 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4575390 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6574392 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 459296719872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9795385660928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9342141802496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1728535 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9342159805440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11125072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 _runtime‚ 243.670244496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6549682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9234302031614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9342141802496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4575392 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 452993023488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1727852 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259998720 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 453225855488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9342159805440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10901373407744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 451860041470 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserÕ»» ved_bytes.small_pool.allocated‚ 528482304 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 452993023488 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6549682 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11125072 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6574392 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9795385660928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8782424950784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1358 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 _step‚18 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9234056029742 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9795134825984 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32ûÏl°&ˆªM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11125072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 _runtime‚ 243.670244496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6549682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9234302031614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9342141802496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4575392 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 452993023488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1727852 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259998720 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 453225855488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9342159805440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10901373407744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 451860041470 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 452993023488 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6549682 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11125072 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6574392 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9795385660928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8782424950784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1358 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 _step‚18 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9234056029742 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9795134825984 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2685323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283938816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4413175 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11360386188800 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8782441990144 trainer/global_step‚249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 451631078958 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4413860 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11149784 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11360670127616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11149784 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4575390 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6574392 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 459296719872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9795385660928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9342141802496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1728535 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9342159805440û-•>0‰+ _wandbruntime‚243  _runtime‚243UÆ ÐPŠÊ /loss/FBt1/detection_loss‚32.075445115566254 trainer/global_step‚249 /loss/FRt4/assignment_loss‚7.420201006802645 1loss/SRqqt1/assignment_loss‚5.383503816565689 1loss/SRqqt3/assignment_loss‚5.215289741265969 /loss/FRt2/assignment_loss‚7.333254247098355 /loss/FRt3/assignment_loss‚7.153195443700572 _step‚19 2loss/SRqqt4/assignment_loss‚5.2332424068450925 0loss/FBt3/assignment_loss‚1.3413577874501545 &loss/total_loss‚143.79270352735386 /loss/FRt1/assignment_loss‚7.272064745426178 0loss/FBt1/assignment_loss‚1.6541723251342773 /loss/FRt1/detection_loss‚3.9667651962593213 .loss/FRt2/detection_loss‚4.668727699969266 /loss/FRt3/detection_loss‚2.3494456985690553 /loss/FRt4/detection_loss‚2.8736161402680658 0loss/SRqqt2/detection_loss‚5.455756601691246 _runtime‚ 243.671333015 1loss/SRqqt2/assignment_loss‚5.283004220326742 /loss/FBt2/assignment_loss‚0.850814204949599 1loss/SRqqt4/detection_loss‚2.9255166339874266 .loss/FBt3/detection_loss‚14.76907720665137 epoch‚0 $ _timestamp‚1.788395180111643e+09 0loss/SRqqt1/detection_loss‚2.626531147084149 /loss/FBt2/detection_loss‚14.104894851262753 0loss/FBt4/assignment_loss‚0.6110397552100706 1loss/SRqqt3/detection_loss‚1.9563935037366436 /loss/FBt4/detection_loss‚1.2693940315329597ú(€T‹Î 1loss/SRqqt2/assignment_loss‚5.283004220326742 /loss/FBt2/assignment_loss‚0.850814204949599 1loss/SRqqt4/detection_loss‚2.9255166339874266 .loss/FBt3/detection_loss‚14.76907720665137 epoch‚0 $ _timestamp‚1.788395180111643e+09 0loss/SRqqt1/detection_loss‚2.626531147084149 /loss/FBt2/detection_loss‚14.104894851262753 0loss/FBt4/assignment_loss‚0.6110397552100706 1loss/SRqqt3/detection_loss‚1.9563935037366436 /loss/FBt4/detection_loss‚1.2693940315329597 /loss/FBt1/detection_loss‚32.075445115566254 trainer/global_step‚249 /loss/FRt4/assignment_loss‚7.420201006802645 1loss/SRqqt1/assignment_loss‚5.383503816565689 1loss/SRqqt3/assignment_loss‚5.215289741265969 /loss/FRt2/assignment_loss‚7.333254247098355 /loss/FRt3/assignment_loss‚7.153195443700572 _step‚19 2loss/SRqqt4/assignment_loss‚5.2332424068450925 0loss/FBt3/assignment_loss‚1.3413577874501545 &loss/total_loss‚143.79270352735386 /loss/FRt1/assignment_loss‚7.272064745426178 0loss/FBt1/assignment_loss‚1.6541723251342773 /loss/FRt1/detection_loss‚3.9667651962593213 .loss/FRt2/detection_loss‚4.668727699969266 /loss/FRt3/detection_loss‚2.3494456985690553 /loss/FRt4/detection_loss‚2.8736161402680658 0loss/SRqqt2/detection_loss‚5.455756601691246 _runtime‚ 243.671333015¥–a'œŒ‚(:‘ ·ýâÔÉÓøÒ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.086' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.417199999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̆²;8‚(:­ ·ýâÔĈÈÕ disk.rbd0.in‚0 disk.dm-0.in‚ 0.33203125 disk.sda.out‚ 137.4609375 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 137.4609375 disk.rbd1.out‚ 0.06298828125 disk.nvme0c0n1.in‚ 0.33203125 disk.nvme0n1.in‚ 0.33203125$ memory_percent‚93.83080005645752 network.recv‚181900 disk.rbd2.out‚15.21533203125 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 137.4609375 cpu‚25.03748573164984 disk.sda1.out‚0% disk.nvme0n1.out‚1359.80126953125" disk.dm-0.out‚1359.80126953125) proc.memory.availableMB‚ 1010.76171875# proc.memory.rssMB‚ 2747.58984375* proc.memory.percent‚16.769957542419434 network.sent‚370157% disk./.usageGB‚1065.8967094421387 disk.sda.in‚ 0.06640625 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda3.in‚ 0.06640625 disk.rbd0.out‚0 disk.rbd3.out‚0) disk./.usagePercent‚71.53322251757072 disk.rbd1.in‚0' disk.nvme0c0n1.out‚1359.80126953125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.in‚0î&oÓŽ‚(:… ÆýâÔÃðÝÒ gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.253' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8506 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ï3r/‚(:¤ ÆýâÔ“ä›Ö disk.dm-0.in‚ 0.33203125 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚1529.7353515625 disk.sda.out‚ 140.37109375* proc.memory.percent‚16.769957542419434 disk.rbd3.out‚0 disk.sda2.in‚0& disk.nvme0c0n1.out‚1529.5947265625# proc.memory.availableMB‚1010.75 proc.cpu.threads‚17! disk.dm-0.out‚1529.7353515625) disk./.usagePercent‚71.57375030229369 disk.rbd2.out‚16.95751953125 disk.dm-1.out‚ 140.37109375 disk.nvme0n1.in‚ 0.33203125 disk.sda.in‚ 0.06640625 cpu‚25.038006045725 disk.drbd1777.in‚0 disk.rbd3.in‚0$ disk./.usageGB‚1066.500602722168 disk.sda2.out‚0$ memory_percent‚93.83087158203125# proc.memory.rssMB‚ 2747.58984375 network.recv‚188154 disk.sda3.out‚ 140.37109375 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125 disk.nvme0c0n1.in‚ 0.33203125 disk.rbd0.out‚0 network.sent‚420364 disk.rbd2.in‚0 disk.dm-1.in‚ 0.06640625<éO‚(:… ÕýâÔžëÒ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.7102 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0† 5‘‚(:ª ÕýâÔä×Æ× disk.dm-1.out‚ 141.890625 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.038358182223856 network.sent‚426287 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.06298828125" disk.dm-0.out‚4827.11279296875 disk.sda1.out‚0 proc.memory.rssMB‚ 2747.59375 network.recv‚192400 disk.rbd1.in‚0$ memory_percent‚93.83254051208496 proc.cpu.threads‚17 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 15.29296875' disk.nvme0c0n1.out‚4827.11279296875 disk.rbd0.out‚0 disk.dm-0.in‚ 15.29296875 disk.sda.in‚ 0.06640625( proc.memory.availableMB‚ 1010.4765625 disk.rbd2.in‚0% disk./.usageGB‚1071.6365356445312 disk.sda3.in‚ 0.06640625 disk.dm-1.in‚ 0.06640625 disk.nvme0n1.in‚ 15.29296875 disk.sda.out‚ 141.890625 disk.rbd3.out‚0 disk.sda3.out‚ 141.890625% disk.nvme0n1.out‚4827.11279296875* proc.memory.percent‚16.769981384277344) disk./.usagePercent‚71.91842707004825 disk.rbd2.out‚18.46923828125 disk.rbd3.in‚0ÖÑTÀ0’+ _wandbruntime‚291  _runtime‚291øÓ“Æ“À ' lr-AdamW/pg2‚5.5154011065168036e-05 trainer/global_step‚299 % _timestamp‚1.7883952276278727e+09 _runtime‚ 291.187805435 _step‚20 ' lr-AdamW/pg1‚5.5154011065168036e-05ÈGÊ”Ä ' lr-AdamW/pg2‚5.5154011065168036e-05 trainer/global_step‚299 % _timestamp‚1.7883952276278727e+09 _runtime‚ 291.187805435 _step‚20 ' lr-AdamW/pg1‚5.5154011065168036e-05¬çJ0•+ _wandbruntime‚291  _runtime‚291CRk¼¯'–©O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13296870 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11162132011520 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 541888528384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2065793 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 _step‚21 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 541655709696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5272411 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3207320 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11703769718272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11704020539904 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7830709 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 540256571030 _runtime‚ 291.189006518 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 541888528384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10493832926208 trainer/global_step‚299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11162114008576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7855393 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 277660672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 253720576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466163 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11034106536598 % _timestamp‚1.7883952276288502e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5466161 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13026916042752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13575674181632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466161 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11703769718272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1358 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13575951842304 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 548758138880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7830709 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 541655709696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11704020539904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466163 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11162132011520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3207322 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2065091 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 549011859456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11033860534934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11703769718272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11704020539904 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7830709 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 540256571030 _runtime‚ 291.189006518 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 541888528384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10493832926208 trainer/global_step‚299 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11162114008576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7855393 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ß× ´! 232818688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 277660672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 253720576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5466163 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11034106536598 % _timestamp‚1.7883952276288502e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5466161 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13026916042752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13575674181632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466161 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11703769718272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1358 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13575951842304 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 548758138880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7830709 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 541655709696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11704020539904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5466163 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11162132011520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3207322 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2065091 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 549011859456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11033860534934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 541655709696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5272411 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3207320 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13365836 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 % _timestamp‚1.7883952285425267e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10528759619584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 543697690112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11199256298496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2072623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288133120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11070590961750 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10528776658944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 _step‚22 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5289889 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13341124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1358 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2071916 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5484340 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5484340 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13621166612992 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 550847720448sÀ ÷½&š·M TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5289889 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13341124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1358 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2071916 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5484340 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5484340 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13621166612992 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 550847720448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 542060304678 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13620878479872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7856784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7856784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 543464858112 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 trainer/global_step‚299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13365836 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13341124 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5484342 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11742721156608 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 543464858112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13365836 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 % _timestamp‚1.7883952285425267e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10528759619584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 543697690112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11199256298496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2072623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288133120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11070590961750 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10528776658944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 _step‚22 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚716 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 632127544320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2412127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 340.988968773 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9137811 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15537608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13651356048896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚349 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9137811 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13019246507520 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 632360363008 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6375113 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13651356048896 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 632360363008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 640285835264 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12870395466942 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 632127544320 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 640543750144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 632360363008 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6375113 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13651356048896 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 632360363008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 640285835264 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12870395466942 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 632127544320 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 640543750144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚716 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 632127544320 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2412127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 340.988968773 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9137811 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15537608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13651356048896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚349 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9137811 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13019246507520 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3750625 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2419554 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 632260567886 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15581888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 % _timestamp‚1.7883952784176447e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6393290 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6393292 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2418848 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12275111327744 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 633936692736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9163886 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3750623 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12907125893758 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 634169524736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15239264377344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15239240437248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6169471 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 634169524736 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6393290 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9163886 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13690558322176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15581888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 642370968064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6393292 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13690307487232 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13690558322176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13056370794496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12275094288384 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15557176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9188596 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1290©¿‘^a7371895630 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13056388797440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15881635345408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13056370794496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚26ÙÅÖ"½&¬·M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3750625 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2419554 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 632260567886 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15581888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 % _timestamp‚1.7883952784176447e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6393290 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6393292 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2418848 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12275111327744 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 633936692736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9163886 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3750623 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12907125893758 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 634169524736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15239264377344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15239240437248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6169471 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 634169524736 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6393290 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9163886 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13690558322176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15581888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 642370968064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6393292 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13690307487232 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13690558322176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13056370794496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12275094288384 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15557176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9188596 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12907371895630 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13056388797440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15881635345408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13056370794496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚26 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15881353503744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710pÚé0­+ _wandbruntime‚341  _runtime‚341‹Ç¹´O®É 2loss/SRqqt2/assignment_loss‚3.5464330383988676 1loss/FBt1/assignment_loss‚0.37972688401008353 /loss/FRt3/detection_loss‚3.7906535239663066 1loss/SRqqt1/detection_loss‚3.1348503817174564 1loss/SRqqt4/detection_loss‚2.5554401419611406 %loss/total_loss‚96.90705125169245 % _timestamp‚1.7883952784204154e+09 .loss/FRt2/assignment_loss‚7.39470454454422 0loss/FBt4/assignment_loss‚0.6928734570209469 _step‚27 /loss/FRt4/detection_loss‚3.6013537408865015 0loss/SRqqt2/detection_loss‚2.572308062774236 .loss/FBt1/detection_loss‚9.383966330219717 .loss/FBt3/detection_loss‚4.502463041527851 epoch‚0 .loss/FRt2/detection_loss‚4.237786031514406 trainer/global_step‚349 _runtime‚ 341.980145098 2loss/SRqqt4/assignment_loss‚3.6158587777096294 1loss/FBt2/assignment_loss‚0.22315638702362775 /loss/FRt1/assignment_loss‚7.263789936348244 2loss/SRqqt3/assignment_loss‚3.2309392778961747 .loss/FRt4/assignment_loss‚7.33065885046254 0loss/FBt3/assignment_loss‚0.3156211263172933 .loss/FRt1/detection_loss‚5.412816963813923 /loss/FRt3/assignment_loss‚7.181766824024479 0loss/SRqqt3/detection_loss‚2.280040448723378 .loss/FBt2/detection_loss‚7.065632872581482 /loss/FBt4/detection_loss‚3.3419218364905516 0loss/SRqqt1/assignment_loss‚3.85228877175938šžS¯Í .loss/FBt1/detection_loss‚9.383966330219717 .loss/FBt3/detection_loss‚4.502463041527851 epoch‚0 .loss/FRt2/detection_loss‚4.237786031514406 trainer/global_step‚349 _runtime‚ 341.980145098 2loss/SRqqt4/assignment_loss‚3.6158587777096294 1loss/FBt2/assignment_loss‚0.22315638702362775 /loss/FRt1/assignment_loss‚7.263789936348244 2loss/SRqqt3/assignment_loss‚3.2309392778961747 .loss/FRt4/assignment_loss‚7.33065885046254 0loss/FBt3/assignment_loss‚0.3156211263172933 .loss/FRt1/detection_loss‚5.412816963813923 /loss/FRt3/assignment_loss‚7.181766824024479 0loss/SRqqt3/detection_loss‚2.280040448723378 .loss/FBt2/detection_loss‚7.065632872581482 /loss/FBt4/detection_loss‚3.3419218364905516 0loss/SRqqt1/assignment_loss‚3.85228877175938 2loss/SRqqt2/assignment_loss‚3.5464330383988676 1loss/FBt1/assignment_loss‚0.37972688401008353 /loss/FRt3/detection_loss‚3.7906535239663066 1loss/SRqqt1/detection_loss‚3.1348503817174564 1loss/SRqqt4/detection_loss‚2.5554401419611406 %loss/total_loss‚96.90705125169245 % _timestamp‚1.7883952784204154e+09 .loss/FRt2/assignment_loss‚7.39470454454422 0loss/FBt4/assignment_loss‚0.6928734570209469 _step‚27 /loss/FRt4/detection_loss‚3.6013537408865015 0loss/SRqqt2/detection_loss‚2.572308062774236>"%°‚(:š ‘þâÔÀ““Ö disk.sda1.in‚0 disk.sda3.out‚161.25 network.recv‚212095 disk.dm-0.in‚ 167.40625 disk.sda.out‚161.25 disk.sda2.in‚0 disk.dm-1.out‚161.25$ memory_percent‚93.82944107055664 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 167.40625' disk.nvme0c0n1.out‚10854.0048828125 network.sent‚494702 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.06640625 disk.rbd2.in‚0 disk.rbd1.out‚ 0.06298828125) disk./.usagePercent‚72.11486933129684% disk./.usageGB‚1074.5636672973633% disk.nvme0n1.out‚10854.0048828125 disk.rbd2.out‚21.82470703125 disk.dm-1.in‚ 0.06640625# proc.memory.rssMB‚ 2747.59765625" disk.dm-0.out‚10854.0048828125 disk.rbd3.out‚0 disk.sda3.in‚ 0.06640625' proc.memory.availableMB‚ 1010.984375 cpu‚25.040434609295882 disk.nvme0n1.in‚ 167.40625 disk.drbd1777.out‚0* proc.memory.percent‚16.770005226135254ê;£±‚(:… ‘þâÔ…ÈÖ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.596' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1192 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öµ 8²‚(:­  þâÔ¡¡ÌÕ% disk.nvme0n1.out‚10859.6650390625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 cpu‚25.041526618779837 network.recv‚217522 disk.nvme0n1.in‚ 167.40625 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 162.37109375 disk.nvme0c0n1.in‚ 167.40625 disk.dm-0.in‚ 167.40625 disk.drbd1777.in‚0 disk.rbd2.out‚21.99267578125 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.sda2.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚72.11445664715166 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 162.37109375 disk.rbd1.out‚ 0.06298828125$ disk./.usageGB‚1074.557518005371 disk.dm-1.out‚ 162.37109375 disk.sda.in‚ 0.06640625 disk.sda1.out‚0 disk.rbd3.out‚0 network.sent‚544267" disk.dm-0.out‚10859.6650390625$ memory_percent‚93.83251667022705) proc.memory.availableMB‚ 1010.48046875# proc.memory.rssMB‚ 2747.59765625' disk.nvme0c0n1.out‚10859.6650390625 disk.sda1.in‚0* proc.memory.percent‚16.770005226135254ã[1&›³‚(:  þâÔ’ëÙÕ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.177000000000003 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03{‘›´‚(: ¯þâÔ¹ÜïÒ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚96.549' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.309800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GzÅ4µ‚(:© ¯þâÔôöÛÕ disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 180.5390625 disk.dm-0.in‚ 167.40625" disk.dm-0.out‚10964.9619140625 disk.sda.in‚ 0.06640625 network.recv‚221377) disk./.usagePercent‚72.11459386719001' disk.nvme0c0n1.out‚10964.9619140625% disk.nvme0n1.out‚10964.9619140625 network.sent‚549778 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.06298828125" proc.memory.rssMB‚ 2747.6015625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda3.out‚ 180.5390625 disk.rbd0.out‚0% disk./.usageGB‚1074.5595626831055 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.sda.out‚ 180.5390625* proc.memory.percent‚16.770029067993164 cpu‚25.041827644859957 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda3.in‚ 0.06640625 disk.nvme0n1.in‚ 167.40625$ memory_percent‚93.82948875427246 disk.rbd2.out‚23.50439453125 disk.rbd3.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 167.40625( proc.memory.availableMB‚ 1010.9765625û´¬U›¶‚(: ¾þâÔÿ½“Ó gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.531' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.906200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05¥¢¸+·‚(:  ¾þâÔõÿåÕ% disk./.usageGB‚1074.5593757629395 disk.dm-1.in‚ 0.06640625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 181.2109375 disk.rbd1.in‚0* proc.memory.percent‚16.770029067993164 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 11064.1796875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚25.18017578125 disk.rbd1.out‚ 0.06298828125 network.sent‚556511 disk.rbd0.out‚0) disk./.usagePercent‚72.11458132282084 disk.sda.out‚ 181.2109375 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 167.40625 disk.dm-0.out‚ 11064.1796875 disk.sda.in‚ 0.06640625 disk.sda2.out‚0 disk.rbd0.in‚0" disk.nvme0n1.out‚ 11064.1796875 disk.sda2.in‚0 disk.nvme0n1.in‚ 167.40625 disk.dm-0.in‚ 167.40625 disk.drbd1777.out‚0 disk.sda3.in‚ 0.06640625$ memory_percent‚93.82948875427246" proc.memory.rssMB‚ 2747.6015625 cpu‚25.042713926200282 network.recv‚226549 disk.dm-1.out‚ 181.2109375 disk.drbd1777.in‚0( proc.memory.availableMB‚ 1010.9765625&ü€ 0¸+ _wandbruntime‚392  _runtime‚392õà4:ù½ _runtime‚ 392.964073861 _step‚28 & lr-AdamW/pg1‚7.360016861204697e-05 & lr-AdamW/pg2‚7.360016861204697e-05 trainer/global_step‚399 $ _timestamp‚1.788395329404162e+09aïJǺÁ $ _timestamp‚1.788395329404162e+09 _runtime‚ 392.964073861 _step‚28 & lr-AdamW/pg1‚7.360016861204697e-05 & lr-AdamW/pg2‚7.360016861204697e-05 trainer/global_step‚399‰á ²0»+ _wandbruntime‚392  _runtime‚392á{*˜²'¼¬O XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15599193201152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7284061 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17728974 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17364807012352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17364830952448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2758406 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15599193201152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 731880522752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10444913 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4272622 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13986502263808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17753660 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚29 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10469597 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13986519303168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14876343000576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 255817728 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 720428135142 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 trainer/global_step‚399 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 732136340480 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 720657097446 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 722599378944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14876361003520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4272620 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7284063 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 722599378944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 722832197632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7031714 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15598942379520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788395329405021e+09 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18096967292928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14876343000576 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7284063 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7284061 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14876361003520 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15598942379520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10469597 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18096687535104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7031026 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2759092 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 722599378944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 JBDeviceStatsMonitor.on_train°ø]ä_batch_start/segment.small_pool.current‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 722832197632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7031714 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15598942379520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788395329405021e+09 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18096967292928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14876343000576 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7284063 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7284061 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14876361003520 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15598942379520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10469597 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18096687535104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7031026 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2759092 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7284061 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17728974 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17364807012352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17364830952448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2758406 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15599193201152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 731880522752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10444913 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4272622 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13986502263808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17753660 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚29 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10469597 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13986519303168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14876343000576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 255817728 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 720428135142 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252÷q0¾+ _wandbruntime‚393  _runtime‚393ý7‡¼&¿¶M I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7302240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14913503293440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14913485290496 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7302240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17408209862144 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 722231868582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17773228 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14743660825766 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14913503293440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17797940 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7302242 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10495698 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7302242 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 724641359360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14743906827638 $ _timestamp‚1.788395330416637e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14913485290496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10470988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 724408527360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14021428957184 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 724408527360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4283275 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17408185922048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚30 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4283273 _runtime‚ 393.977177135 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18141896952832 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17797940 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 733966835200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 722460831094 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7049519 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2765541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 279744512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10470988 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7048814 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10495698 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15637893817856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893D e À&ÀºM _runtime‚ 393.977177135 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18141896952832 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17797940 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1376 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 733966835200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 722460831094 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7049519 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2765541 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 279744512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10470988 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7048814 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10495698 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15637893817856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7302240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14913503293440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14913485290496 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7302240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17408209862144 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 722231868582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17773228 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14743660825766 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14913503293440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17797940 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7302242 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10495698 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7302242 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 724641359360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14743906827638 $ _timestamp‚1.788395330416637e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14913485290496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10470988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 724408527360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14021428957184 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 724408527360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4283275 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17408185922048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚30 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4283273LOÎÚ0Á+ _wandbruntime‚393  _runtime‚393ëwIÂà /loss/FRt4/assignment_loss‚7.346566394523338 /loss/FRt1/detection_loss‚3.6852034053641667 .loss/FBt1/detection_loss‚9.938454965129495 %loss/total_loss‚95.33551536692256 $ _timestamp‚1.788395330418719e+09 /loss/FRt3/assignment_loss‚7.234955084525932 0loss/FBt1/assignment_loss‚0.4356773995095864 1loss/FBt3/assignment_loss‚0.42123943019722554 -loss/FBt3/detection_loss‚5.58014995938745 1loss/SRqqt3/detection_loss‚2.1481375786604295 0loss/SRqqt4/detection_loss‚3.158753983334655 1loss/SRqqt2/assignment_loss‚3.512628955356145 1loss/SRqqt3/assignment_loss‚3.396419484039833 1loss/SRqqt2/detection_loss‚2.6261064693079157 /loss/FRt2/assignment_loss‚7.428001750599254 .loss/FRt2/detection_loss‚2.984787893566218 .loss/FRt4/detection_loss‚2.929254555729804 /loss/FBt4/detection_loss‚1.4959568067271425 epoch‚0 _step‚31 1loss/SRqqt4/assignment_loss‚3.901126660798725 .loss/FRt3/detection_loss‚2.277788487531371 trainer/global_step‚399 0loss/FBt4/assignment_loss‚0.8385857862810935 .loss/FRt1/assignment_loss‚7.21652923311506 0loss/SRqqt1/assignment_loss‚3.70154337088267 1loss/FBt2/assignment_loss‚0.19677812123554758 /loss/SRqqt1/detection_loss‚4.19355363089107 .loss/FBt2/detection_loss‚8.687315960228442 _runtime‚ 393.978489999¯½'MÃÇ .loss/FRt4/detection_loss‚2.929254555729804 /loss/FBt4/detection_loss‚1.4959568067271425 epoch‚0 _step‚31 1loss/SRqqt4/assignment_loss‚3.901126660798725 .loss/FRt3/detection_loss‚2.277788487531371 trainer/global_step‚399 0loss/FBt4/assignment_loss‚0.8385857862810935 .loss/FRt1/assignment_loss‚7.21652923311506 0loss/SRqqt1/assignment_loss‚3.70154337088267 1loss/FBt2/assignment_loss‚0.19677812123554758 /loss/SRqqt1/detection_loss‚4.19355363089107 .loss/FBt2/detection_loss‚8.687315960228442 _runtime‚ 393.978489999 /loss/FRt4/assignment_loss‚7.346566394523338 /loss/FRt1/detection_loss‚3.6852034053641667 .loss/FBt1/detection_loss‚9.938454965129495 %loss/total_loss‚95.33551536692256 $ _timestamp‚1.788395330418719e+09 /loss/FRt3/assignment_loss‚7.234955084525932 0loss/FBt1/assignment_loss‚0.4356773995095864 1loss/FBt3/assignment_loss‚0.42123943019722554 -loss/FBt3/detection_loss‚5.58014995938745 1loss/SRqqt3/detection_loss‚2.1481375786604295 0loss/SRqqt4/detection_loss‚3.158753983334655 1loss/SRqqt2/assignment_loss‚3.512628955356145 1loss/SRqqt3/assignment_loss‚3.396419484039833 1loss/SRqqt2/detection_loss‚2.6261064693079157 /loss/FRt2/assignment_loss‚7.428001750599254 .loss/FRt2/detection_loss‚2.984787893566218—†ò ›Ä‚(: ÍþâÔâú¡Ó gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.005' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.400999999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÍØ7Å‚(:¬ ÍþâÔý÷ÙÕ cpu‚25.042976522645194 disk.nvme0c0n1.in‚ 167.40625' disk.nvme0c0n1.out‚11066.5048828125% disk.nvme0n1.out‚11066.5048828125 disk.sda2.out‚0$ memory_percent‚93.83018016815186) proc.memory.availableMB‚ 1010.86328125) disk./.usagePercent‚72.11460692357426 disk.dm-0.in‚ 167.40625 disk.sda3.in‚ 0.06640625* proc.memory.percent‚16.770029067993164$ disk./.usageGB‚1074.559757232666 disk.dm-1.out‚ 193.94140625 disk.nvme0n1.in‚ 167.40625 disk.sda.in‚ 0.06640625 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2747.6015625" disk.dm-0.out‚11066.5048828125 disk.sda.out‚ 193.94140625 disk.sda1.in‚0 disk.rbd3.in‚0 network.sent‚562319 network.recv‚230770 disk.dm-1.in‚ 0.06640625 disk.sda1.out‚0 disk.rbd2.out‚25.18017578125 disk.sda3.out‚ 193.94140625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 0.06298828125¡<õk›Æ‚(: ÜþâÔÓÆ’Ó gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.005' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.400999999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žì°ÆðÇ‚(:© ÜþâÔ­þœÖ disk.sda3.in‚ 0.06640625' disk.nvme0c0n1.out‚11164.8056640625$ memory_percent‚93.83032321929932 network.recv‚237230 disk.rbd0.out‚0 cpu‚25.043151621229253 disk.sda.in‚ 0.06640625 dûFDisk.sda1.in‚0 disk.sda3.out‚ 208.5078125 disk.dm-0.in‚ 167.40625 disk.sda.out‚ 208.5078125 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.06298828125 disk.nvme0n1.in‚ 167.40625) proc.memory.availableMB‚ 1010.83984375 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2747.6015625 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 167.40625) disk./.usagePercent‚72.11460564353659" disk.dm-0.out‚11164.8056640625 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 208.5078125 disk.rbd0.in‚0% disk.nvme0n1.out‚11164.8056640625* proc.memory.percent‚16.770029067993164 proc.cpu.threads‚17 network.sent‚612637% disk./.usageGB‚1074.5597381591797 disk.rbd2.out‚ 26.8720703125 disk.rbd3.in‚0 disk.sda2.out‚0<ïžÈ‚(:… ëþâÔƒãÒ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚97.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4972 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë“}4É‚(:© ëþâÔÉúµÖ disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 221.97265625 proc.cpu.threads‚17% disk./.usageGB‚1074.5597839355469 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 0.06298828125$ memory_percent‚93.83058547973633# proc.memory.rssMB‚ 2747.60546875 disk.drbd1777.out‚0* proc.memory.percent‚16.770052909851074 cpu‚25.043353660435173 disk.dm-1.out‚ 221.97265625 disk.rbd0.out‚0% disk.nvme0n1.out‚11165.5908203125 disk.sda.in‚ 0.06640625 disk.rbd2.out‚ 26.8720703125 disk.rbd3.in‚0' disk.nvme0c0n1.out‚11165.5908203125 disk.dm-0.in‚ 167.40625" disk.dm-0.out‚11165.5908203125 network.sent‚618321' disk./.usagePercent‚72.114608715627 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 221.97265625 disk.nvme0c0n1.in‚ 167.40625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 167.40625 network.recv‚241413 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625' proc.memory.availableMB‚ 1010.796875RC‡‡0Ê+ _wandbruntime‚445  _runtime‚445aZæÚÄ˾ trainer/global_step‚449 % _timestamp‚1.7883953818041792e+09 _runtime‚ 445.364009829 _step‚32 & lr-AdamW/pg1‚8.282324738548645e-05 & lr-AdamW/pg2‚8.282324738548645e-05Ÿ-«‘ÈÌ & lr-AdamW/pg2‚8.282324738548645e-05 trainer/global_step‚449 % _timestamp‚1.7883953818041792e+09 _runtime‚ 445.364009829 _step‚32 & lr-AdamW/pg1‚8.282324738548645e-05 (u™B0Í+ _wandbruntime‚445  _runtime‚445Dïƒà²'άO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 823736567808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19945026 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20357220664320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15732853971968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8193013 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8193011 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11776699 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17546528710144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7910791 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19533776437248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16543711332622 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 813304032256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 trainer/global_step‚449 $ _timestamp‚1.788395381804969e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16733457496576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 810628398350 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 813071213568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4805272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7911480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 813304032256 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 813071213568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3105521 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292340736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19533752497152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16543465330958 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3106208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15732836932608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8193013 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268400640 _runtime‚ 445.365042875 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 810857360654 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11776699 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11752015 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16733475499520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16733475499520 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19969712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8193011 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11752015 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17546779531776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 823468167168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20357513005056 _step‚33 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 810628398350 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 813071213568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4805272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7911480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 813304032256 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 813071213568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3105521 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292340736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19533752497152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16543465330958 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3106208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15732836932608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8193013 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 268400640 _runtime‚ 445.365042875 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 810857360654 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11776699 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11752015 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16733475499520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16733475499520 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19969712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8193011 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11752015 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17546779531776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 823468167168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20357513005056 _step‚33 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17546528710144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7910791 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19533776437248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16543711332622 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 813304032256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 trainer/global_step‚449 $ _timestamp‚1.788395381804969e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16733457496576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 814880361984 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8211190 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20402431845888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19577155346944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚449 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17585730983424 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17585480148480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19577131406848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20013992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3113112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883953828164904e+09 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19989280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19989280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8211192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20402713687552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4815923 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 446.377456794 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16770599786496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 _step‚34 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 825300439040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16770617789440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7928342 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17585730983424 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3112419 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 815113193984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7929037 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11778090 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ËÑÒ––11778090 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 814880361984 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8211190 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚20402431845888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19577155346944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚449 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17585730983424 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17585480148480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚19577131406848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20013992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3113112 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883953828164904e+09 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19989280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19989280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8211192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20402713687552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4815923 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 446.377456794 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16770599786496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 _step‚34 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 825300439040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16770617789440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7928342 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17585730983424 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3112419 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0"›a®0Ó+ _wandbruntime‚446  _runtime‚446öa3ßRÔÌ 0loss/FRt4/assignment_loss‚7.1894289617953095 1loss/SRqqt2/detection_loss‚2.3663022469912396 1loss/SRqqt4/detection_loss‚1.8900340410955323 epoch‚0 1loss/FBt1/assignment_loss‚0.21359795621699756 1loss/FBt2/assignment_loss‚0.30169945736881343 0loss/FBt3/assignment_loss‚0.3724815070818295 0loss/FBt4/assignment_loss‚0.5014804811992993 .loss/FRt1/detection_loss‚3.961686258316056 trainer/global_step‚449 /loss/FRt1/assignment_loss‚7.183721046447754 .loss/FRt4/detection_loss‚3.608756132747816 0loss/SRqqt3/detection_loss‚4.305036059288082 /loss/FRt2/detection_loss‚2.4726877210364826 /loss/FBt1/detection_loss‚17.495911290248234 -loss/FBt3/detection_loss‚4.97881432576105 %loss/total_loss‚99.63656836097795 2loss/SRqqt1/assignment_loss‚3.7532291412353516 1loss/SRqqt1/detection_loss‚3.0398846545342115 1loss/SRqqt4/assignment_loss‚3.570924147963524 .loss/FBt2/detection_loss‚6.629448254555464 % _timestamp‚1.7883953828193207e+09 _runtime‚ 446.379083218 _step‚35 /loss/FRt2/assignment_loss‚7.276336186174033 /loss/FBt4/detection_loss‚1.3696458935846698 2loss/SRqqt2/assignment_loss‚3.4308300448245688 1loss/SRqqt3/assignment_loss‚3.289298063999898 .loss/FRt3/detection_loss‚3.160905928133016 /loss/FRt3/assignment_loss‚7.274428560378704ã%ÎîVÕÐ 2loss/SRqqt1/assignment_loss‚3.7532291412353516 1loss/SRqqt1/detection_loss‚3.0398846545342115 1loss/SRqqt4/assignment_loss‚3.570924147963524 .loss/FBt2/detection_loss‚6.629448254555464 % _timestamp‚1.7883953828193207e+09 _runtime‚ 446.379083218 _step‚35 /loss/FRt2/assignment_loss‚7.276336186174033 /loss/FBt4/detection_loss‚1.3696458935846698 2loss/SRqqt2/assignment_loss‚3.4308300448245688 1loss/SRqqt3/assignment_loss‚3.289298063999898 .loss/FRt3/detection_loss‚3.160905928133016 /loss/FRt3/assignment_loss‚7.274428560378704 0loss/FRt4/assignment_loss‚7.1894289617953095 1loss/SRqqt2/detection_loss‚2.3663022469912396 1loss/SRqqt4/detection_loss‚1.8900340410955323 epoch‚0 1loss/FBt1/assignment_loss‚0.21359795621699756 1loss/FBt2/assignment_loss‚0.30169945736881343 0loss/FBt3/assignment_loss‚0.3724815070818295 0loss/FBt4/assignment_loss‚0.5014804811992993 .loss/FRt1/detection_loss‚3.961686258316056 trainer/global_step‚449 /loss/FRt1/assignment_loss‚7.183721046447754 .loss/FRt4/detection_loss‚3.608756132747816 0loss/SRqqt3/detection_loss‚4.305036059288082 /loss/FRt2/detection_loss‚2.4726877210364826 /loss/FBt1/detection_loss‚17.495911290248234 -loss/FBt3/detection_loss‚4.97881432576105 %loss/total_loss‚99.63656836097795#:¿úÖ‚(:… úþâÔ¦Þ‚Ó gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.596' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1192 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïâ•â2ׂ(:§ úþâÔûíõÕ network.sent‚625012 disk.sda2.out‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚11263.927734375 disk.nvme0n1.in‚ 167.40625 disk.sda.in‚ 0.06640625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 222.76953125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 167.40625 disk.rbd0.out‚0' proc.memory.availableMB‚ 1010.734375# proc.memory.rssMB‚ 2747.60546875* proc.memory.percent‚16.770052909851074 disk.rbd0.in‚0 disk.dm-0.in‚ 167.40625) disk./.usagePercent‚72.11460743558933% disk./.usageGB‚1074.5597648620605 disk.sda1.in‚0 cpu‚25.04414125531065$ disk.nvme0n1.out‚11263.927734375 network.recv‚246108 disk.rbd1.out‚ 0.06298828125 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 222.76953125$ memory_percent‚93.83096694946289 disk.sda.out‚ 222.76953125 proc.cpu.threads‚17 disk.sda1.out‚0! disk.dm-0.out‚11263.927734375 disk.rbd2.out‚ 28.4619140625Ú)Ø‚(:… ‰ÿâÔ‡†çÒ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3862 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÌÌ¢Ù‚(:‹ ‰ÿâÔ´ÎíÕ network.sent‚674333 network.recv‚251399 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 167.40625( proc.memory.availableMB‚ 1010.9765625# proc.memory.rssMB‚ 2747.60546875 disk.dm-0.out‚11362.87890625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 0.06298828125) disk./.usagePercent‚72.11460999566467% disk./.usageGB‚1074.5598030090332 disk.sda1.out‚0$ memory_percent‚93.82948875427246* proc.memory.percent‚16.770052909851074 disk.sda3.out‚235 disk.rbd1.in‚0# disk.nvme0n1.out‚11362.87890625 proc.cpu.threads‚17 disk.sda3.in‚ 0.06640625 disk.sda.out‚235 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.dm-0.in‚ 167.40625% disk.nvme0c0n1.out‚11362.87890625 disk.rbd3.in‚0 disk.sda.in‚ 0.06640625 disk.rbd2.out‚ 28.4853515625 disk.nvme0c0n1.in‚ 167.40625 disk.dm-1.out‚235 disk.rbd0.in‚0 cpu‚25.044834980412457Od€ŽÚ‚(:ƒ ˜ÿâÔ±íéÒ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.82' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.964 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö‹2M1Û‚(:¦ ˜ÿâÔÙܪÖ disk.dm-1.out‚ 248.69921875 disk.sda.out‚ 248.69921875 disk.rbd2.in‚0 disk.sda3.out‚ 248.69921875 network.sent‚681051 disk.dm-1.in‚ 0.06640625 disk.rbd3.in‚0% disk.nvme0c0n1.out‚11363.22265625) proc.memory.availableMB‚ 1010.72265625% disk./.usageGB‚1074.5599632263184 disk.rbd0.out‚0 disk.dm-0.out‚11363.22265625 disk.sda.in‚ 0.06640625 disk.drbd1777.out‚0 disk.sda2.in‚0 network.recv‚256274# disk.nvme0n1.out‚11363.22265625 disk.rbd3.out‚0 cpu‚25.04493904103742 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda2.out‚0$ memory_percent‚93.83103847503662 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 167.40625* proc.memory.percent‚16.770052909851074( disk./.usagePercent‚72.1146207479811 disk.dm-0.in‚ 167.40625 disk.rbd2.out‚30.20849609375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2747.60546875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 167.40625 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 0.06298828125ÇjŽÜ‚(:ƒ §ÿâÔÆÇ´Ó gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚75.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]iz&Ý‚(:› §ÿâÔ¯¼ÁÖ disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.dm-1.in‚ 0.06640625# memory_percent‚93.8295841217041$ disk./.usageGB‚1074.559642791748 disk.sda3.out‚ 249.7890625# disk.nvme0c0n1.out‚ 11461.578125 disk.dm-0.out‚ 11461.578125 cpu‚25.04506611833555 network.sent‚686548 network.recv‚260150 disk.sda.out‚ 249.7890625 disk.drbd1777.in‚0 disk.rbd2.out‚31.81005859375 disk.rbd3.out‚0 disk.dm-1.out‚ 249.7890625 disk.dm-0.in‚ 167.40625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 167.40625( proc.memory.availableMB‚ 1010.9609375# proc.memory.rssMB‚ 2747.61328125 disk.sda.in‚ 0.06640625 disk.rbd2.in‚0! disk.nvme0n1.out‚ 11461.578125) disk./.usagePercent‚72.11459924334824 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.06298828125 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 167.40625* proc.memory.percent‚16.770100593566895 proc.cpu.threads‚17 disk.rbd3.in‚0¯ÞrA0Þ+ _wandbruntime‚497  _runtime‚497 $#²Äß¾ & lr-AdamW/pg1‚9.204632615892591e-05 & lr-AdamW/pg2‚9.204632615892591e-05 trainer/global_step‚499 % _timestamp‚1.7883954344304829e+09 _runtime‚ 497.990312734 _step‚36ªãúÈà & lr-AdamW/pg1‚9.204632615892591e-05 & lr-AdamW/pg2‚9.204632615892591e-05 trainer/global_step‚499 % _timestamp‚1.7883954344304829e+09 _runtime‚ 497.990312734 _step‚36$@D(0á+ _wandbruntime‚497  _runtime‚497–D .³'â­O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 286049280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9101961 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 901057623862 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 % _timestamp‚1.7883954344313436e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22161078 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3454136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚499 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9101963 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13059117 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 900828661558 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 _runtime‚ 497.991379495 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19494365862400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚17479188640768 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5337920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19494115040768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18590589995520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 915046713344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13083801 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13083801 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13059117 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19494365862400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9101961 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18590571992576 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 903543048192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21702721922048 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3453397 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚37 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5337922 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18380000262966 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚17479171601408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22185764 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚741 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18590589995520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18380246264630 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19494115040768 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 903775866880 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8791317 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22617744695296 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 915308822528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8792058 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18590571992576 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 903543048192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21702721922048 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3453397 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚37 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5337922 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18380000262966 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚17479171601408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22185764 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚741 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 MGDeviceStatsMonitor.on_train_batchåÓå_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18590589995520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚18380246264630 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19494115040768 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 903775866880 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8791317 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22617744695296 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 915308822528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8792058 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18627732285440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18627732285440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚38 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 % _timestamp‚1.7883954354007895e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3461296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22205332 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21746100831744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 902861357510 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22663239412224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18416976691654 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9120140 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18416730689782 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22662957570560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8809871 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22230044 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 trainer/global_step‚499 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22205332 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8809157 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 917138580480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 916880678912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9120142±›YˆÁ&æ»M J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8809157 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 917138580480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 916880678912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9120142 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19533317314048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18627714282496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚17514098294784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚18627732285440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18627732285440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚38 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 % _timestamp‚1.7883954354007895e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3461296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22205332 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21746100831744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 902861357510 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22663239412224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚18416976691654 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9120140 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18416730689782 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22662957570560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8809871 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22230044 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 trainer/global_step‚499 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22205332 EÜ0ó+ _wandbruntime‚550  _runtime‚550~‚Rc¸'ô²O UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19225506270208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14366219 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9670786 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9670092 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3800214 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 _step‚41 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5870570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 994247701504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24401816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21441701371392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23871643466752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20447686488576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5870572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23871667406848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20216535194974 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 _runtime‚ 550.575993337 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14390903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976YH‚Ù¼'õ¶O ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 994247701504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24401816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21441701371392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23871643466752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20447686488576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5870572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23871667406848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20216535194974 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 _runtime‚ 550.575993337 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14390903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19225506270208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14366219 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9670786 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9670092 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3800214 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 _step‚41 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5870570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072)Gî7k0ö+ _wandbruntime‚551  _runtime‚551ÊÎýçÂ&÷¼M NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14392294 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9687818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20484828778496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14417004 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1376 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14417004 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20484846781440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9688516 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3807291 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23915046316544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 993061620718 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚21480903644672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10029090 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚21480652809728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 996056863232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 995824031232 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24446096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 trainer/global_step‚549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1008475326976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 % _timestamp‚1.7883954880337758e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 _step‚42i÷\ÌUøÀM WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24923497703424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24446096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10029092 J·Û/Vq%=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14392294 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 trainer/global_step‚549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1008475326976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 % _timestamp‚1.7883954880337758e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 _step‚42 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14392294 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9687818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20484828778496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14417004 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1376 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14417004 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20484846781440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9688516 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3807291 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23915046316544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 993061620718 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚21480903644672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10029090 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚21480652809728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 996056863232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 995824031232 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24446096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10919863 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1084719536128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15673321 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26617868 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10919861 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10549214 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27139084329472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15698005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22304800984576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6403222 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26040612891648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 251623424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20971840939008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1098219814400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10919863 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1081229187974 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1084486717440 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1084486717440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23389287702016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15673321 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22304800984576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚45 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23389538523648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22304818987520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4145994 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23389287702016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4146697 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10919861 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1084719536128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22053070126982Q2ˆd‰¿O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15698005 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22304800984576 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚703 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6403222 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26040612891648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 251623424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20971840939008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1098219814400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10919863 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1081229187974 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1084486717440 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1084486717440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBpž©faDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23389287702016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15673321 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22304800984576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚45 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23389538523648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22304818987520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4145994 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23389287702016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4146697 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10919861 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1084719536128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22053070126982 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26593182 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22053316128646 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10919863 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1084719536128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15673321 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26617868 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10919861 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10549214 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27139084329472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307-Övb0Š+ _wandbruntime‚603  _runtime‚603A?°7Ë&‹ÅM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1086295865856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26662148 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1086528697856 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚23428489975296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15724106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27184014317056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10566925 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27184296158720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26662148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21006784671744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26083991801344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22090046555670 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22341961277440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10938042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10567639 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6413875 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1083032921414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1086528697856 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10938042 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6413873 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 % _timestamp‚1.7883955399084163e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15699396 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22341943274496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1086295865856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559РusÏ&ŒÉM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15699396 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1086528697856 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚23428489975296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15724106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27184014317056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10566925 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27184296158720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26662148 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21006784671744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26083991801344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22090046555670 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22341961277440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10938042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10567639 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6413875 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1083032921414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1086528697856 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10938042 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6413873 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 % _timestamp‚1.7883955399084163e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15699396 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22341943274496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1086295865856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1086295865856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26662148 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 E“‚(:³ ®€ãÔÛÀßÖ% disk./.usageGB‚1083.2375450134277 disk.rbd2.out‚40.27685546875 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 334.48046875 proc.cpu.threads‚17 disk.rbd1.in‚0$ memory_percent‚93.83137226104736 network.sent‚872994 disk.rbd2.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 334.48046875# proc.memory.rssMB‚ 2747.62109375 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda3.in‚ 0.0703125 disk.dm-0.in‚ 334.48046875( disk./.usagePercent‚72.6969805426901 disk.sda2.in‚0 disk.dm-1.in‚ 0.0703125 disk.dm-1.out‚ 394.07421875) proc.memory.availableMB‚ 1010.66796875 cpu‚25.047205907999942 disk.rbd0.out‚0 network.recv‚304461 disk.sda1.out‚0 disk.sda3.out‚ 394.07421875' disk.nvme0c0n1.out‚21068.3818359375% disk.nvme0n1.out‚21068.3818359375* proc.memory.percent‚16.770148277282715" disk.dm-0.out‚21068.3818359375 disk.sda.in‚ 0.0703125 disk.sda.out‚ 394.07421875 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.36962890625¡§”‚(:… ½€ãÔÛÛùÒ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9418 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áš^m•‚(:¦ ½€ãÔৼÕ disk.sda3.out‚395.8125 disk.nvme0n1.in‚ 334.48046875& disk.nvme0n1.out‚½Å5Ä21143.36474609375 disk.sda.in‚ 0.0703125 disk.sda1.out‚0) proc.memory.availableMB‚ 1010.66015625 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚21143.36474609375 disk.drbd1777.in‚0 network.sent‚878793 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.sda.out‚395.8125 disk.sda1.in‚0 disk.rbd2.out‚40.30029296875 disk.sda3.in‚ 0.0703125( disk.nvme0c0n1.out‚21143.36474609375* proc.memory.percent‚16.770172119140625 network.recv‚309051 disk.dm-1.out‚395.8125$ memory_percent‚93.83141994476318 cpu‚25.047366072755608 proc.cpu.threads‚17 disk.dm-0.in‚ 334.48046875 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.36962890625 disk.rbd0.in‚0 disk.drbd1777.out‚0 proc.memory.rssMB‚2747.625) disk./.usagePercent‚72.69587382211957% disk./.usageGB‚1083.2210540771484 disk.rbd1.in‚0 disk.dm-1.in‚ 0.0703125 disk.rbd0.out‚0À)ºž0–+ _wandbruntime‚655  _runtime‚655“КÆ—À ' lr-AdamW/pg1‚0.00011971556247924432 ' lr-AdamW/pg2‚0.00011971556247924432 trainer/global_step‚649 % _timestamp‚1.7883955917057576e+09 _runtime‚ 655.265603969 _step‚48-¾ªíÊ˜Ä ' lr-AdamW/pg1‚0.00011971556247924432 ' lr-AdamW/pg2‚0.00011971556247924432 trainer/global_step‚649 % _timestamp‚1.7883955917057576e+09 _runtime‚ 655.265603969 _step‚480HŽ¦0™+ _wandbruntime‚655  _runtime‚655ï¡ŽüÁ'š»O @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11828811 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17005107 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6935870 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1190050887680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1174958552064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28833920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23889851060654 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1171429451182 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1171658413486 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25337124854272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24161915480576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4491655 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28209534436352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264206336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22718175607808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11828811 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28209558376448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 _step‚49 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1175191370752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4490976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1189786681344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29399321117696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25336874032640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28809234 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29399609264128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 % _timestamp‚1.7883955917065895e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16980423 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11828813 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1175191370752 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25336874032640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1174958552064 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25337124854272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24161933483520 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23889605058990 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24161915480576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17005107 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11426846 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4490976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1189786681344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29399321117696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25336874032640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28809234 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29399609264128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 % _timestamp‚1.7883955917065895e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16980423 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚681 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11828813 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1175191370752 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25336874032640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1174958552064 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25337124854272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24161933483520 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23889605058990 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24161915480576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17005107 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11426846 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1174958552064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28833920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23889851060654 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1171429451182 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1171658413486 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25337124854272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24161915480576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4491655 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28209534436352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 264206336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22718175607808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11828811 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28209558376448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 _step‚49 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1175191370752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚35161³¿=‹0œ+ _wandbruntime‚656  _runtime‚656½y¢Ê&ÄM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24199057770496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11444278 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29444819600384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1191620218368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1173462147134 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 % _timestamp‚1.7883955927514617e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28853488 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11846990 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23926335485806 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24199075773440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11846990 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 286035968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1191882314240 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23926581487678 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1176767700480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28878200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4498456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28252913346048 _runtime‚ 656.31218026 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17031208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1173233184622 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11846992'öà†žÈM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4498456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28252913346048 _runtime‚ 656.31218026 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17031208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1173233184622 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 NA&VÚH#DeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11846992 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24199057770496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11444278 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29444819600384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1191620218368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1173462147134 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 % _timestamp‚1.7883955927514617e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28853488 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11846990 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23926335485806 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24199075773440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11846990 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 286035968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1191882314240 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23926581487678 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1176767700480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28878200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 5213972482¢¶00Ÿ+ _wandbruntime‚656  _runtime‚656ÙC5 N È /loss/FRt3/assignment_loss‚7.206813451720447 0loss/FBt3/assignment_loss‚0.6590892224829938 .loss/FRt3/detection_loss‚3.459686433850992 1loss/SRqqt3/detection_loss‚2.0374764176818796 /loss/FBt4/detection_loss‚3.3013562883975225 _step‚51 0loss/FBt2/assignment_loss‚0.5136954626750907 .loss/FRt4/detection_loss‚4.244970332417223 0loss/SRqqt2/detection_loss‚2.030289183947302 .loss/FRt1/detection_loss‚4.240758657455444 /loss/FRt4/assignment_loss‚7.344536172019111 1loss/SRqqt1/assignment_loss‚3.594136262551332 1loss/SRqqt3/assignment_loss‚3.134511547429221 trainer/global_step‚649 % _timestamp‚1.7883955927539582e+09 _runtime‚ 656.314220034 /loss/FRt2/detection_loss‚2.9755680111271365 0loss/SRqqt4/detection_loss‚3.077184917070282 %loss/total_loss‚87.66969917202309 epoch‚0 /loss/FRt1/assignment_loss‚7.265740229533269 0loss/FBt1/assignment_loss‚0.2779417610228328 0loss/SRqqt1/detection_loss‚2.436987492303571 /loss/FBt3/detection_loss‚2.6116611225841324 2loss/SRqqt2/assignment_loss‚3.2570234177604553 .loss/FBt1/detection_loss‚7.286250165530613 0loss/FBt4/assignment_loss‚0.5792248234152794 2loss/SRqqt4/assignment_loss‚3.6142024240995707 /loss/FRt2/assignment_loss‚7.410196604552092 .loss/FBt2/detection_loss‚5.110398770395289nCWïR¡Ì /loss/FRt2/assignment_loss‚7.410196604552092 .loss/FBt2/detection_loss‚5.110398770395289 /loss/FRt3/assignment_loss‚7.206813451720447 0loss/FBt3/assignment_loss‚0.6590892224829938 .loss/FRt3/detection_loss‚3.459686433850992 1loss/SRqqt3/detection_loss‚2.0374764176818796 /loss/FBt4/detection_loss‚3.3013562883975225 _step‚51 0loss/FBt2/assignment_loss‚0.5136954626750907 .loss/FRt4/detection_loss‚4.244970332417223 0loss/SRqqt2/detection_loss‚2.030289183947302 .loss/FRt1/detection_loss‚4.240758657455444 /loss/FRt4/assignment_loss‚7.344536172019111 1loss/SRqqt1/assignment_loss‚3.594136262551332 1loss/SRqqt3/assignment_loss‚3.134511547429221 trainer/global_step‚649 % _timestamp‚1.7883955927539582e+09 _runtime‚ 656.314220034 /loss/FRt2/detection_loss‚2.9755680111271365 0loss/SRqqt4/detection_loss‚3.077184917070282 %loss/total_loss‚87.66969917202309 epoch‚0 /loss/FRt1/assignment_loss‚7.265740229533269 0loss/FBt1/assignment_loss‚0.2779417610228328 0loss/SRqqt1/detection_loss‚2.436987492303571 /loss/FBt3/detection_loss‚2.6116611225841324 2loss/SRqqt2/assignment_loss‚3.2570234177604553 .loss/FBt1/detection_loss‚7.286250165530613 0loss/FBt4/assignment_loss‚0.5792248234152794 2loss/SRqqt4/assignment_loss‚3.61420242409957073k;›¢‚(: Ì€ãÔ´›–Õ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚79.729' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.945799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é@¨½*£‚(:Ÿ Ì€ãÔÝÜÛ× network.recv‚313620) disk./.usagePercent‚72.69588867055657& disk.nvme0n1.out‚21241.70458984375 disk.sda3.out‚396.6875( proc.memory.availableMB‚ 1011.4609375* proc.memory.percent‚16.770172119140625 cpu‚25.04761808467265 disk.nvme0n1.in‚ 334.48046875 disk.drbd1777.out‚0 disk.rbd3.out‚0# disk.dm-0.out‚21241.70458984375 disk.rbd2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0703125 disk.sda3.in‚ 0.0703125 disk.rbd1.out‚ 0.36962890625" disk.nvme0c0n1.in‚ 334.48046875% disk./.usageGB‚1083.2212753295898 disk.drbd1777.in‚0 disk.rbd2.out‚ 41.9609375 disk.sda2.in‚0# memory_percent‚93.8265323638916 network.sent‚885415 disk.dm-0.in‚ 334.48046875 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.sda.out‚396.6875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚396.6875 proc.memory.rssMB‚2747.625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚ 0.0703125 disk.sda1.in‚0( disk.nvme0c0n1.out‚21241.704589843750Ϥ‚(:… Û€ãÔÚîÒ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚89.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9178 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!|0¥‚(:¥ Û€ãÔ¼ÆÛÕ disk.nvme0n1.in‚ 334.48046875 disk.sda.out‚ 398.46484375 disk.drbd1777.out‚0 disk.rbd2.out‚ 41.98828125 disk.rbd1.out‚ 0.36962890625 disk.rbd0.out‚0 disk.dm-0.in‚ 334.48046875 disk.drbd1777.in‚0 disk.rbd2.in‚0( disk./.usagePercent‚72.6068523222382 disk.sda.in‚ 0.0703125 disk.sda1.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 398.46484375 network.sent‚934515 disk.sda2.in‚0 disk.sda3.out‚ 398.46484375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 334.48046875# disk.nvme0n1.out‚21341.30859375 disk.sda1.out‚0 network.recv‚318907 disk.dm-0.out‚21341.30859375) proc.memory.availableMB‚ 1011.20703125 proc.memory.rssMB‚2747.625 proc.cpu.threads‚17% disk./.usageGB‚1081.8945693969727% disk.nvme0c0n1.out‚21341.30859375 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.0703125$ memory_percent‚93.82808208465576 disk.dm-1.in‚ 0.0703125* proc.memory.percent‚16.770172119140625 cpu‚25.047817055945274 disk.sda2.out‚0$מH¦‚(:… ê€ãÔ€¹‰Ó gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚99.658' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9316 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lçk7§‚(:¬ ê€ãÔ•ìÄÕ' proc.memory.availableMB‚ 1011.203125 disk.sda.out‚ 400.62890625 disk.rbd2.in‚0 disk.rbd1.out‚ 0.36962890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 400.62890625 disk.dm-1.in‚ 0.0703125 disk.sda.in‚ 0.0703125 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 cpu‚25.04787541312634 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 334.48046875' disk.nvme0c0n1.out‚21341.6904296875 disk.dm-1.out‚ 400.62890625 disk.nvme0n1.in‚ 334.48046875 disk.rbd1.in‚0$ memory_percent‚93.82810592651367% disk./.usageGB‚1081.8945007324219 disk.rbd2.out‚43.71142578125% disk.nvme0n1.out‚21341.6904296875 disk.dm-0.in‚ 334.48046875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.0703125 proc.memory.rssMB‚2747.625* proc.memory.percent‚16.770172119140625 network.sent‚940831 network.recv‚323280) disk./.usagePercent‚72.60684771410259" disk.dm-0.out‚21341.6904296875 disk.sda1.in‚0 disk.drbd1777.out‚0²œs¨‚(:… ù€ãÔ‰âÓ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.357' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÞvú9©‚(:® ù€ãÔ€áÐÖ network.recv‚327399 disk.dm-1.out‚ 402.859375 disk.nvme0n1.in‚ 334.48046875 disk.sda1.out‚0 disk.rbd2.out‚45.30908203125 disk.rbd3.in‚0) disk./.usagePercent‚72.60685027417793 disk.rbd0.in‚0& disk.nvme0n1.out‚21443.54443359375 disk.dm-0.in‚ 334.48046875 disk.sda.out‚ 402.859375) proc.memory.availableMB‚ 1011.10546875* proc.memory.percent‚16.770648956298828 disk.sda.in‚ 0.0703125 disk.rbd2.in‚0 network.sent‚946332% disk./.usageGB‚1081.8945388793945 disk.dm-1.in‚ 0.0703125 disk.rbd3.out‚0 disk.sda3.out‚ 402.859375 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚93.82870197296143 disk.rbd1.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2747.703125 proc.cpu.threads‚17 disk.sda3.in‚ 0.0703125" disk.nvme0c0n1.in‚ 334.48046875( disk.nvme0c0n1.out‚21443.54443359375 disk.sda2.out‚0 cpu‚25.04818336987102 disk.rbd1.out‚ 0.36962890625# disk.dm-0.out‚21443.54443359375 disk.drbd1777.in‚0 disk.drbd1777.out‚0ñÿ0ª+ _wandbruntime‚708  _runtime‚708"M‹®Ã«½ & lr-AdamW/pg1‚0.0001289386412526838 & lr-AdamW/pg2‚0.0001289386412526838 trainer/global_step‚699 $ _timestamp‚1.788395644843158e+09 _runtime‚ 708.403008008 _step‚52ËʪhǬÁ & lr-AdamW/pg2‚0.0001289386412526838 trainer/global_step‚699 $ _timestamp‚1.788395644843158e+09 _runtime‚ 708.403008008 _step‚52 & lr-AdamW/pg1‚0.00012893864125268384‹¹ :0­+ _wandbruntime‚708  _runtime‚708kšÇfÁ'®»O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30378479921152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26019047979520 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31049972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30378503861248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27284711184896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1261629714390 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18287525 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7468522 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12737763 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1265663205376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12306380 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18287525 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1265663205376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26019047979520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25726139990998 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12737763 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27284460363264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27284711184896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1265430386688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26019029976576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4838564 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4837860 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12737761 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24464510276608 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27284460363264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1261858676694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12307086 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25726385992662 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26019029976576 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30378479921152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26019047979520 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31049972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30378503861248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27284711184896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1261629714390 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18287525 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7468522 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12737763 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1265663205376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12306380 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18287525 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1265663205376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 _LDevºE€ßiceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26019047979520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25726139990998 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12737763 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27284460363264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27284711184896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1265430386688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26019029976576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4838564 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4837860 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12737761 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24464510276608 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27284460363264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1261858676694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12307086 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1348 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25726385992662 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26019029976576 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1348 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26056190269440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31705477497344 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25763116419686 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31069540 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27323662636544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 709.390387842 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31094252 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12755940 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1283594726400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1263662410342 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26056190269440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1267239535104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27323662636544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12755940 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12324513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259998720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1267472367104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31094252 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18338310 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30421882770944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24499436969984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31069540 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12323780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12755942 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26056172266496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7479175 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26056190269440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31705477497344 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25763116419686 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31069540 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27323662636544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 709.390387842 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31094252 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12755940 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1283594726400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1263662410342 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26056190269440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1267239535104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27323662636544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12755940 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12324513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259998720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1267472367104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31094252 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18338310 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30421882770944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24499436969984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31069540 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12323780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12755942 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26056172266496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7479175 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5596åʇê0³+ _wandbruntime‚709  _runtime‚709 ÜãNL´Æ 0loss/SRqqt4/detection_loss‚2.384105946272319 % _timestamp‚1.7883956458323953e+09 /loss/FRt3/assignment_loss‚7.239310552489083 2loss/SRqqt1/assignment_loss‚3.6256137509499826 2loss/SRqqt2/assignment_loss‚3.1808435005300186 /loss/FRt3/detection_loss‚2.6625530186618835 1loss/SRqqt1/detection_loss‚3.3121403518975234 .loss/FBt1/detection_loss‚4.925413358211517 _runtime‚ 709.392171814 -loss/FRt4/detection_loss‚4.08862739318126 /loss/FBt3/detection_loss‚2.4969599468118022 1loss/SRqqt4/assignment_loss‚3.579071260080105 0loss/FBt2/assignment_loss‚0.3808653466626043 1loss/SRqqt2/detection_loss‚1.6389057803351212 0loss/SRqqt3/detection_loss‚1.989109120328477 .loss/FRt4/assignment_loss‚7.37846513696619 /loss/FBt3/assignment_loss‚0.824748824310622 0loss/FBt4/assignment_loss‚0.2924397667908731 _step‚55 /loss/FRt1/assignment_loss‚7.197554349899292 .loss/FRt2/detection_loss‚2.866182352917699 .loss/FBt4/detection_loss‚7.183303150037925 $loss/total_loss‚85.0786538410256 .loss/FRt1/detection_loss‚4.238243645429611 /loss/FRt2/assignment_loss‚7.432668860142048 1loss/SRqqt3/assignment_loss‚3.066275159517924 /loss/FBt2/detection_loss‚2.6480039816423053 epoch‚0 trainer/global_step‚699 1loss/FBt1/assignment_loss‚0.44724928695941346ŒÚS0PµÊ .loss/FRt1/detection_loss‚4.238243645429611 /loss/FRt2/assignment_loss‚7.432668860142048 1loss/SRqqt3/assignment_loss‚3.066275159517924 /loss/FBt2/detection_loss‚2.6480039816423053 epoch‚0 trainer/global_step‚699 1loss/FBt1/assignment_loss‚0.44724928695941346 0loss/SRqqt4/detection_loss‚2.384105946272319 % _timestamp‚1.7883956458323953e+09 /loss/FRt3/assignment_loss‚7.239310552489083 2loss/SRqqt1/assignment_loss‚3.6256137509499826 2loss/SRqqt2/assignment_loss‚3.1808435005300186 /loss/FRt3/detection_loss‚2.6625530186618835 1loss/SRqqt1/detection_loss‚3.3121403518975234 .loss/FBt1/detection_loss‚4.925413358211517 _runtime‚ 709.392171814 -loss/FRt4/detection_loss‚4.08862739318126 /loss/FBt3/detection_loss‚2.4969599468118022 1loss/SRqqt4/assignment_loss‚3.579071260080105 0loss/FBt2/assignment_loss‚0.3808653466626043 1loss/SRqqt2/detection_loss‚1.6389057803351212 0loss/SRqqt3/detection_loss‚1.989109120328477 .loss/FRt4/assignment_loss‚7.37846513696619 /loss/FBt3/assignment_loss‚0.824748824310622 0loss/FBt4/assignment_loss‚0.2924397667908731 _step‚55 /loss/FRt1/assignment_loss‚7.197554349899292 .loss/FRt2/detection_loss‚2.866182352917699 .loss/FBt4/detection_loss‚7.183303150037925 $loss/total_loss‚85.07865384102567o0…e¶‚(:… ˆãÔŠÀ‚Ó gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚90.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0458 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãÿ“q<·‚(:± ˆãÔŸéÝÖ disk.rbd0.out‚0! proc.memory.rssMB‚ 2747.703125 network.recv‚331819 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0703125 proc.cpu.threads‚17 disk.sda3.out‚ 438.8984375& disk.nvme0n1.out‚21443.94287109375 disk.rbd3.in‚0 disk.rbd1.out‚ 0.36962890625 disk.rbd0.in‚0* proc.memory.percent‚16.770648956298828 cpu‚25.04821350700922% disk./.usageGB‚1081.8945007324219 network.sent‚952618 disk.dm-0.in‚ 334.48046875 disk.sda.in‚ 0.0703125) disk./.usagePercent‚72.60684771410259 disk.rbd2.out‚45.44970703125 disk.sda2.in‚0 disk.sda3.in‚ 0.0703125 disk.rbd1.in‚0( disk.nvme0c0n1.out‚21443.94287109375 disk.dm-1.out‚ 438.8984375$ memory_percent‚93.82927417755127 disk.drbd1777.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1011.01171875 disk.nvme0n1.in‚ 334.48046875# disk.dm-0.out‚21443.94287109375 disk.sda.out‚ 438.8984375 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 334.48046875¡>dš¸‚(: —ãÔÄׄÓ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚77.31' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.462000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x¢ãÛ@¹‚(:µ —ãÔ¨¯»Ö disk.nvme0n1.in‚ 334.48046875" disk.dm-0.out‚21542.6962890625% disk./.usageGB‚1081.8945655822754 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 469.33203125 disk.dm-1.out‚ 469.33203125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.0019e+06 disk.drbd1777.in‚0 disk.sda3.in‚ 0.0703125 disk.rbd1.out‚ 0.36962890625" disk.nvme0c0n1.in‚ 334.48046875) disk./.usagePercent‚72.60685206623066 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda.in‚ 0.0703125 disk.sda.out‚ 469.33203125 cpu‚25.04826679135719 proc.cpu.threads‚17 disk.dm-1.in‚ 0.0703125 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd2.out‚46.98486328125' disk.nvme0c0n1.out‚21542.6962890625$ memory_percent‚93.82784366607666 disk.sda1.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚21542.6962890625 disk.dm-0.in‚ 334.48046875) proc.memory.availableMB‚ 1011.24609375! proc.memory.rssMB‚ 2747.703125* proc.memory.percent‚16.770648956298828 network.recv‚337204 disk.sda2.in‚0MÇD“º‚(:… ¦ãÔ„î‘Ó gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Df"À»‚(:´ ¦ãÔ£ÁîÖ) proc.memory.availableMB‚ 1010.98828125 network.recv‚341708% disk.nvme0n1.out‚21640.9501953125 disk.dm-0.in‚ 334.48046875" disk.nvme0c0n1.in‚ 334.48046875 disk.rbd0.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0' disk.nvme0c0n1.out‚21640.9501953125 network.sent‚ 1.008122e+06$ disk./.usageGB‚1081.894546508789 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 cpu‚25.048595176996727 disk.sda3.out‚ 470.16015625 disk.dm-1.out‚ 470.16015625 disk.rbd0.out‚0 disk.sda.out‚ 470.16015625 disk.drbd1777.in‚0$ memory_percent‚93.82941722869873' disk./.usagePercent‚72.606850786193!Uùµ disk.sda2.out‚0 disk.sda3.in‚ 0.0703125 disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚ 47.1376953125! proc.memory.rssMB‚ 2747.703125 disk.rbd1.out‚ 0.36962890625 disk.dm-1.in‚ 0.0703125" disk.dm-0.out‚21640.9501953125 disk.sda.in‚ 0.0703125 disk.sda1.out‚0* proc.memory.percent‚16.770648956298828 disk.nvme0n1.in‚ 334.48046875´U§0¼+ _wandbruntime‚760  _runtime‚760gŒ‘—Ž¿ _step‚56 ' lr-AdamW/pg1‚0.00013816172002612326 ' lr-AdamW/pg2‚0.00013816172002612326 trainer/global_step‚749 $ _timestamp‚1.788395697305512e+09 _runtime‚ 760.865345532A,ösɾà ' lr-AdamW/pg1‚0.00013816172002612326 ' lr-AdamW/pg2‚0.00013816172002612326 trainer/global_step‚749 $ _timestamp‚1.788395697305512e+09 _runtime‚ 760.865345532 _step‚568°©Þ0¿+ _wandbruntime‚760  _runtime‚760ßT0À'ÀºO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1373361966592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19619311 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1351829977598 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8001172 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1352058939902 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13646713 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29232046693888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33920533651968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13186745 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13186054 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13646711 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26210844945408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19619311 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27876162475520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29232046693888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 277660672 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33266024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1373108246016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 trainer/global_step‚749 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29232297515520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 $ _timestamp‚1.788395697306358e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19594627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33920811312640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29232297515520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27562674923006 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13646711 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8001170 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13646713 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32547425405952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚57 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29232046693888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 277660672 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33266024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1373108246016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 trainer/global_step‚749 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29232297515520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 $ _timestamp‚1.788395697306358e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19594627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33920811312640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29232297515520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27562674923006 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13646711 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8001170 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13646713 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32547425405952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚57 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33285592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚749 _runtime‚ 761.848441819 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33285592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27913286762496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5191510 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1357711369728 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33966026088960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26245788678144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1353862673550 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27913304765440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13664890 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13664892 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19620702 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 _step‚58 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13204020 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27913286762496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266290176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13203333 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8011825 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27599405349822 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29271248967168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27913304765440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚32590804315648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1353633711038 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27599651351694 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5192195 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32590828255744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1357944201728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33310304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13664892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33965735858688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944}~Q‘ÎÄÉM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1353862673550 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27913304765440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13664890 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13664892 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ÊhY!L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19620702 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 _step‚58 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13204020 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27913286762496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 266290176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13203333 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8011825 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27599405349822 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29271248967168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27913304765440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚32590804315648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1353633711038 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27599651351694 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5192195 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚32590828255744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1357944201728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33310304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13664892 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33965735858688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33285592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚749 _runtime‚ 761.848441819 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33285592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27913286762496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5191510 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1357711369728 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33966026088960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26245788678144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304:ut0Å+ _wandbruntime‚761  _runtime‚761½æS6RÆÌ 2loss/SRqqt3/assignment_loss‚3.1478771780218398 1loss/SRqqt4/assignment_loss‚3.466098969632929 1loss/FBt1/assignment_loss‚0.46826690379409763 1loss/FBt4/assignment_loss‚0.34181409208880115 1loss/SRqqt1/detection_loss‚2.5910156442698566 0loss/FBt2/assignment_loss‚0.3443741636393468 .loss/FRt3/detection_loss‚2.040489244798664 /loss/FRt1/detection_loss‚3.9168335080146788 %loss/total_loss‚85.10097965087043 % _timestamp‚1.7883956982898276e+09 _step‚59 .loss/FRt2/detection_loss‚2.379022555855604 epoch‚0 /loss/FBt3/detection_loss‚1.9767978274138933 .loss/FBt4/detection_loss‚9.026619578401247 .loss/FBt1/detection_loss‚4.994395844895264 .loss/FRt2/assignment_loss‚7.41047073510977 /loss/FRt4/assignment_loss‚7.336260381734596 _runtime‚ 761.849585344 /loss/FRt1/assignment_loss‚7.114620518684387 2loss/SRqqt1/assignment_loss‚3.5790556498936246 /loss/FRt4/detection_loss‚2.8440549424234427 1loss/SRqqt2/detection_loss‚1.8631342073683352 0loss/SRqqt3/detection_loss‚2.385987678381908 1loss/SRqqt4/detection_loss‚2.5351248992721893 trainer/global_step‚749 0loss/FBt3/assignment_loss‚0.6240348611803105 .loss/FBt2/detection_loss‚4.096075958299939 0loss/FRt3/assignment_loss‚7.2020759508013725 1loss/SRqqt2/assignment_loss‚3.416478356894325È8âVÇÐ 1loss/SRqqt2/assignment_loss‚3.416478356894325 2loss/SRqqt3/assignment_loss‚3.1478771780218398 1loss/SRqqt4/assignment_loss‚3.466098969632929 1loss/FBt1/assignment_loss‚0.46826690379409763 1loss/FBt4/assignment_loss‚0.34181409208880115 1loss/SRqqt1/detection_loss‚2.5910156442698566 0loss/FBt2/assignment_loss‚0.3443741636393468 .loss/FRt3/detection_loss‚2.040489244798664 /loss/FRt1/detection_loss‚3.9168335080146788 %loss/total_loss‚85.10097965087043 % _timestamp‚1.7883956982898276e+09 _step‚59 .loss/FRt2/detection_loss‚2.379022555855604 epoch‚0 /loss/FBt3/detection_loss‚1.9767978274138933 .loss/FBt4/detection_loss‚9.026619578401247 .loss/FBt1/detection_loss‚4.994395844895264 .loss/FRt2/assignment_loss‚7.41047073510977 /loss/FRt4/assignment_loss‚7.336260381734596 _runtime‚ 761.849585344 /loss/FRt1/assignment_loss‚7.114620518684387 2loss/SRqqt1/assignment_loss‚3.5790556498936246 /loss/FRt4/detection_loss‚2.8440549424234427 1loss/SRqqt2/detection_loss‚1.8631342073683352 0loss/SRqqt3/detection_loss‚2.385987678381908 1loss/SRqqt4/detection_loss‚2.5351248992721893 trainer/global_step‚749 0loss/FBt3/assignment_loss‚0.6240348611803105 .loss/FBt2/detection_loss‚4.096075958299939 0loss/FRt3/assignment_loss‚7.2020759508013725;ÂåÊî›È‚(: µãÔܾÅÓ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.243' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.848599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&@@É‚(:µ µãÔÌ·¼Õ disk.sda3.out‚ 501.80859375$ memory_percent‚93.83001327514648 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.out‚ 48.6767578125 disk.sda2.in‚0 disk.sda3.in‚ 0.0703125 disk.nvme0n1.in‚ 334.48046875 disk.dm-0.in‚ 334.48046875* proc.memory.percent‚16.770648956298828 disk.rbd3.out‚0' proc.memory.availableMB‚ 1010.890625! proc.memory.rssMB‚ 2747.703125' disk./.usagePercent‚72.606850786193 disk.drbd1777.in‚0 disk.sda.in‚ 0.0703125 disk.rbd2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.dm-1.in‚ 0.0703125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 501.80859375 disk.rbd0.in‚0# disk.dm-0.out‚21641.43603515625 disk.sda.out‚ 501.80859375 network.sent‚ 1.013545e+06$ disk./.usageGB‚1081.894546508789 disk.sda1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚21641.43603515625& disk.nvme0n1.out‚21641.43603515625 network.recv‚345574 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.36962890625 cpu‚25.048712801818247Üí`ð›Ê‚(: ÄãÔœˆ§Ó gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚96.421' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.284200000000002 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ÁR£5Ë‚(:ª ÄãÔãÿ€×% disk./.usageGB‚1081.8945655822754 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 334.48046875 disk.sda.in‚ 0.0703125 cpu‚25.04863744389298" disk.nvme0c0n1.in‚ 334.48046875 disk.rbd0.in‚0 disk.rbd2.out‚ 48.8212890625 disk.rbd1.out‚ 0.36962890625( disk.nvme0c0n1.out‚21739.74072265625 disk.dm-1.out‚533.5625& disk.nvme0n1.out‚21739.74072265625 disk.dm-0.in‚ 334.48046875 disk.sda.out‚533.5625' proc.memory.availableMB‚ 1010.609375) disk./.usagePercent‚72.60685206623066 disk.rbd2.in‚0 disk.sda3.in‚ 0.0703125 network.recv‚351080 disk.rbd3.in‚0 disk.sda2.in‚0# disk.dm-0.out‚21739.74072265625* proc.memory.percent‚16.770648956298828 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.sda3.out‚533.5625 disk.rbd1.in‚0$ memory_percent‚93.83172988891602 disk.sda2.out‚0 network.sent‚ 1.06353e+06 disk.rbd3.out‚0 disk.dm-1.in‚ 0.0703125! proc.memory.rssMB‚ 2747.703125 disk.sda1.in‚0 disk.drbd1777.out‚0¤ÝˆH›Ì‚(: ÓãÔì“àÓ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.071199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ÒCÍ‚(:¸ ÓãÔ§Â¸Ö disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 334.48046875 disk.sda.out‚ 561.98828125# memory_percent‚93.8302755355835) proc.memory.availableMB‚ 1010.84765625* proc.memory.percent‚16.770648956298828 disk.rbd2.out‚ 50.3642578125 disk.sda2.out‚0 disk.sda3.out‚ 561.98828125 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 disk.dm-1.in‚ 0.0703125 disk.nvme0n1.in‚ 334.48046875 disk.sda1.out‚0 network.sent‚ 1.068685e+06 network.recv‚354681% disk./.usageGB‚1081.8946075439453 disk.drbd1777.out‚0 disk.dm-1.out‚ 561.98828125 proc.cpu.threads‚17) disk./.usagePercent‚72.60685488231354 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚21838.30419921875! proc.memory.rssMB‚ 2747.703125 cpu‚25.04869796200645 disk.rbd3.out‚0 disk.sda3.in‚ 0.0703125" disk.nvme0c0n1.in‚ 334.48046875 disk.rbd0.out‚0 disk.sda.in‚ 0.0703125& disk.nvme0n1.out‚21838.30419921875# disk.dm-0.out‚21838.30419921875ðÖÕr›Î‚(: âãÔ†ñÝÓ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚79.729' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.945799999999998 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö àšEÏ‚(:º âãÔÆÝÁÖ disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 1010.5859375 proc.cpu.threads‚17) disk./.usagePercent‚72.60686179451696 disk.dm-1.in‚ 0.0703125# proc.memory.rssMB‚ 2747.70703125 network.sent‚ 1.075127e+06% disk./.usageGB‚1081.8947105407715 disk.nvme0n1.in‚ 334.48046875 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 562.83203125 disk.rbd0.in‚0 network.recv‚359155 disk.sda1.in‚0 disk.sda3.out‚ 562.83203125 disk.rbd1.out‚ 0.36962890625$ memory_percent‚93.83187294006348 disk.drbd1777.out‚0& disk.nvme0n1.out‚21838.62841796875 disk.sda.in‚ 0.0703125 disk.sda1.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.dm-0.in‚ 334.48046875# disk.dm-0.out‚21838.62841796875 disk.sda2.in‚0) proc.memory.percent‚16.77067279815674( disk.nvme0c0n1.out‚21838.62841796875 cpu‚25.04901559434919 disk.rbd2.out‚52.03662109375 disk.sda3.in‚ 0.0703125 disk.sda.out‚ 562.83203125 disk.drbd1777.in‚0J$10Ð+ _wandbruntime‚813  _runtime‚813±i–ŠÆÑÀ ' lr-AdamW/pg1‚0.00014738479879956273 ' lr-AdamW/pg2‚0.00014738479879956273 trainer/global_step‚799 % _timestamp‚1.7883957503150518e+09 _runtime‚ 813.874932717 _step‚60AÚ­aÊÒÄ _step‚60 ' lr-AdamW/pg1‚0.00014738479879956273 ' lr-AdamW/pg2‚0.00014738479879956273 trainer/global_step‚799 % _timestamp‚1.7883957503150518e+09 _runtime‚ 813.874932717<å ¯Š0Ó+ _wandbruntime‚813  _runtime‚813šd,%Á'Ô»O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34716394830848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14555663 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 813.876063298 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14555661 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31179633024512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5534824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 251623424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29733276971520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34716370890752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1446374055936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1464924696064 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35457390 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31179883846144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29733276971520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29399209855014 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29733258968576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8533820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35482076 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14555663 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7883957503158925e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20926413 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 trainer/global_step‚799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8533822 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5534120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29399455856678 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14068646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20926413 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27957179614208 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1446606874624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36181043963392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1442259203110 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1446374055936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1442030240806 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29733258968576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1464673072640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31179883846144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14067940 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20901729 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31179633024512 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20901729 ‹¾TÕ¿O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29733276971520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34716370890752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1446374055936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1464924696064 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35457390 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31179883846144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29733276971520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29399209855014 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29733258968576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8533820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35482076 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14555663 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7883957503158925e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20926413 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 trainer/global_step‚799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8533822 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.cŒ<"«6urrent‚24684 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5534120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29399455856678 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14068646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20926413 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27957179614208 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1446606874624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36181043963392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1442259203110 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1446374055936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1442030240806 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29733258968576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1464673072640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31179883846144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14067940 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20901729 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31179633024512 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20901729 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34716394830848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14555663 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 813.876063298 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14555661 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31179633024512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5534824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 251623424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559=½2I0Ö+ _wandbruntime‚814  _runtime‚814¡|æË&×ÅM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29770401258496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36226250530304 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29435940281830 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8544473 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 279744512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1466500729856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31218835297792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1448416036352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5542012 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1444062936758 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31218584462848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1448416036352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29436186283702 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35501644 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1448183204352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34759749800448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 814.905148045 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20952514 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35501644 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20927804 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5541322 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36226530274816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29770401258496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 trainer/global_step‚799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20927804 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 255804416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14573840 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14573842 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35526356 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27992106307584 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14085795 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27992123346944ü˜‘Ï&ØÉM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29770401258496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 trainer/global_step‚799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20927804 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 255804416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14573840 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14573842 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35526356 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27992106307584 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14085795 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27992123346944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29770401258496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36226250530304 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29435940281830 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8544473 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 279744512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1466500729856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31218835297792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1448416036352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5542012 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1444062936758 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31218584462848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1448416036352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29436186283702 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35501644 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1448183204352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34759749800448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 814.905148045 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20952514 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35501644 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20927804 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5541322 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36226530274816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0>mS-q0Ù+ _wandbruntime‚814  _runtime‚814“øn{QÚË .loss/FBt4/detection_loss‚7.921177297830582 %loss/total_loss‚85.68335400876379 epoch‚0 -loss/FBt2/assignment_loss‚1.0251819493439 /loss/FRt4/detection_loss‚3.7503647297620772 _step‚63 2loss/SRqqt4/assignment_loss‚3.5608393601008825 1loss/SRqqt3/detection_loss‚0.8765562278655682 .loss/FBt3/detection_loss‚4.260452375467072 /loss/FRt1/assignment_loss‚6.659610271453857 /loss/FRt3/assignment_loss‚7.193689588914838 .loss/FRt1/detection_loss‚2.652252420783043 0loss/SRqqt4/detection_loss‚2.954124505345219 /loss/FBt2/detection_loss‚3.8264920724523197 0loss/FRt2/assignment_loss‚7.3936117337300225 /loss/FRt4/assignment_loss‚6.908971524238586 1loss/SRqqt3/assignment_loss‚3.124340897653161 1loss/FBt1/assignment_loss‚0.23733865572422794 1loss/FBt3/assignment_loss‚0.46531485289423946 0loss/SRqqt1/detection_loss‚4.732559150265109 .loss/FRt3/detection_loss‚2.284403280600121 % _timestamp‚1.7883957513465595e+09 2loss/SRqqt2/assignment_loss‚3.0203906215470413 1loss/SRqqt2/detection_loss‚2.3306449180597375 _runtime‚ 814.906353266 trainer/global_step‚799 2loss/SRqqt1/assignment_loss‚3.4361354458716606 /loss/FRt2/detection_loss‚2.8249483668746858 .loss/FBt1/detection_loss‚3.808920102344977 1loss/FBt4/assignment_loss‚0.43503365964085483I^‹ŸUÛÏ 2loss/SRqqt4/assignment_loss‚3.5608393601008825 1loss/SRqqt3/detection_loss‚0.8765562278655682 .loss/FBt3/detection_loss‚4.260452375467072 /loss/FRt1/assignment_loss‚6.659610271453857 /loss/FRt3/assignment_loss‚7.193689588914838 .loss/FRt1/detection_loss‚2.652252420783043 0loss/SRqqt4/detection_loss‚2.954124505345219 /loss/FBt2/detection_loss‚3.8264920724523197 0loss/FRt2/assignment_loss‚7.3936117337300225 /loss/FRt4/assignment_loss‚6.908971524238586 1loss/SRqqt3/assignment_loss‚3.124340897653161 1loss/FBt1/assignment_loss‚0.23733865572422794 1loss/FBt3/assignment_loss‚0.46531485289423946 0loss/SRqqt1/detection_loss‚4.732559150265109 .loss/FRt3/detection_loss‚2.284403280600121 % _timestamp‚1.7883957513465595e+09 2loss/SRqqt2/assignment_loss‚3.0203906215470413 1loss/SRqqt2/detection_loss‚2.3306449180597375 _runtime‚ 814.906353266 trainer/global_step‚799 2loss/SRqqt1/assignment_loss‚3.4361354458716606 /loss/FRt2/detection_loss‚2.8249483668746858 .loss/FBt1/detection_loss‚3.808920102344977 1loss/FBt4/assignment_loss‚0.43503365964085483 .loss/FBt4/detection_loss‚7.921177297830582 %loss/total_loss‚85.68335400876379 epoch‚0 -loss/FBt2/assignment_loss‚1.0251819493439 /loss/FRt4/detection_loss‚3.7503647297620772 _step‚63?ó®ÉƒÜ‚(:… ñãÔ—–äÒ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IIepAÝ‚(:¶ ñãÔÑÒÍÕ cpu‚25.049088864582554% disk./.usageGB‚1081.8947105407715 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.0703125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2747.70703125 disk.rbd3.out‚0 network.sent‚ 1.080488e+06 network.recv‚363156 disk.sda.out‚ 597.6171875 disk.dm-1.in‚ 0.0703125 disk.rbd0.out‚0) proc.memory.availableMB‚ 1010.77734375 disk.rbd2.out‚52.06396484375 disk.rbd2.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚72.60686179451696" disk.nvme0c0n1.in‚ 334.48046875 disk.nvme0n1.in‚ 334.48046875" disk.dm-0.out‚21937.1552734375 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 597.6171875% disk.nvme0n1.out‚21937.1552734375 disk.dm-0.in‚ 334.48046875$ memory_percent‚93.83070468902588) proc.memory.percent‚16.77067279815674 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 597.6171875 proc.cpu.threads‚17 disk.sda.in‚ 0.0703125 disk.rbd1.out‚ 0.36962890625' disk.nvme0c0n1.out‚21937.1552734375 disk.rbd0.in‚0Àö“Þ‚(:… €‚ãÔôùÓ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚88.876' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7752 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n~˜«7ß‚(:¬ €‚ãÔÒÇÔÖ disk.dm-0.in‚ 334.48046875 disk.sda.in‚ 0.0703125) proc.memory.percent‚16.77069664001465% disk./.usageGB‚1081.8948669433594 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚22035.4716796875 disk.dm-1.out‚ 617.59375$ memory_percent‚93.82953643798828 network.sent‚ 1.130485e+06 disk.rbd2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.dm-1.in‚ 0.0703125 disk.nvme0n1.in‚ 334.48046875 disk.drbd1777.in‚0& proc.memory.availableMB‚ 1010.96875 network.recv‚368556) disk./.usagePercent‚72.60687229082588 disk.sda3.out‚ 617.59375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚53.66943359375 disk.rbd1.out‚ 0.36962890625 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚22035.4716796875 disk.sda3.in‚ 0.0703125 disk.sda.out‚ 617.59375" proc.memory.rssMB‚ 2747.7109375 cpu‚25.049330240187917 disk.rbd3.out‚0% disk.nvme0n1.out‚22035.4716796875™D`›à‚(: ‚ãÔ›°îÒ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚86.697' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.339399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾i°Dá‚(:¹ ‚ãÔ™à¢Õ disk.rbd0.in‚0 disk.nvme0n1.in‚ 334.48046875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.0703125 disk.rbd3.out‚0" proc.memory.rssMB‚ 2747.7109375 disk.rbd1.out‚ 0.36962890625 disk.sda1.in‚0) proc.memory.percent‚16.77069664001465 disk.rbd2.in‚0 disk.dm-1.out‚ 634.80078125& disk.nvme0n1.out‚22036.04052734375 disk.dm-0.in‚ 334.48046875 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1010.94921875 disk.sda3.in‚ 0.0703125 disk.rbd0.out‚0 disk.rbd2.out‚53.68896484375$ memory_percent‚93.82965564727783 cpu‚25.049423263166897 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 334.48046875 disk.sda.out‚ 634.80078125$ disk./.usageGB‚1081.894931793213# disk.dm-0.out‚22036.04052734375 disk.drbd1777.in‚0) disk./.usagePercent‚72.60687664295395 disk.sda3.out‚ 634.80078125 disk.rbd1.in‚0 disk.dm-1.in‚ 0.0703125 network.sent‚ 1.13482e+06 network.recv‚371676( disk.nvme0c0n1.out‚22036.04052734375¸À0â+ _wandbruntime‚867  _runtime‚867aÞ eã½ & lr-AdamW/pg1‚0È€Ó ©.0001566078775730022 & lr-AdamW/pg2‚0.0001566078775730022 trainer/global_step‚849 $ _timestamp‚1.788395803714264e+09 _runtime‚ 867.274155331 _step‚64¾³&œÇäÁ & lr-AdamW/pg1‚0.0001566078775730022 & lr-AdamW/pg2‚0.0001566078775730022 trainer/global_step‚849 $ _timestamp‚1.788395803714264e+09 _runtime‚ 867.274155331 _step‚64@j3¦0å+ _wandbruntime‚867  _runtime‚867>öï&À'æºO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33127470176768 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33127219355136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31590373464576 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5878579 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14945049 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266303488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31590391467520 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37698128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1556311363072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31235990788686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1537078709248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37673442 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15464613 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33127470176768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36885316375552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15464613 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38441917982208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15464611 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31235744787022 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1532459466318 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 trainer/global_step‚849 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1532230504014 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 $ _timestamp‚1.788395803715178e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1536845890560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22233515 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9066472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31590391467520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31590373464576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36885340315648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 867.275459189 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304àò#Ä'ç¾O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1536845890560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22233515 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9066472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31590391467520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31590373464576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36885340315648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 867.275459189 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚671 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33127470176768 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33127219355136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31590373464576 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5878579 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14945049 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 266303488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31590391467520 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37698128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1556311363072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31235990788686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1537078709248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37673442 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15464613 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33127470176768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36885316375552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15464613 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38441917982208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15464611 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31235744787022 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1532459466318 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 trainer/global_step‚849 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1532230504014 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 $ _timestamp‚1.788395803715178e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252A/Ï@<0è+ _wandbruntime‚868  _runtime‚868Ôïh Ë&éÅM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31627533757440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 868.320035689 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15482792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5885170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 286035968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37742408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1558419231232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1558157135360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37717696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14962968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1538655038976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22234906 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31272475213838 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15482790 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38486852420608 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1538887870976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37742408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29738440976384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31627515754496 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31272721215710 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9077125 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9077123 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1538655038976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22259616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5885843 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29738458015744 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14962293 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33166421628416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 _step‚66 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22259616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36928695285248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 trainer/global_step‚849“ôØÚ©êÉM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31627533757440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15482792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22234906 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1538887870976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37742408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29738440976384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31627515754496 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31272721215710 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 ZMDevÈÐf^&iceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9077125 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9077123 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1538655038976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22259616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5885843 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29738458015744 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14962293 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33166421628416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 _step‚66 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22259616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36928695285248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 trainer/global_step‚849 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31627533757440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 868.320035689 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15482792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5885170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 286035968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37742408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1558419231232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1558157135360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37717696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14962968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1538655038976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22234906 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31272475213838 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15482790 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38486852420608 Bó‚(:³ ¼‚ãÔË­ˆÖ) disk./.usagePercent‚72.60687049877313 disk.sda.in‚ 0.0703125 disk.sda.out‚ 674.41796875 disk.sda1.in‚0 disk.rbd2.out‚53.93603515625) proc.memory.percent‚16.77074432373047 proc.cpu.threads‚17' disk.nvme0c0n1.out‚22233.2607421875 disk.dm-1.out‚ 674.41796875 disk.dm-0.in‚ 334.48046875 disk.drbd1777.out‚0 disk.rbd2.in‚0% disk./.usageGB‚1081.8948402404785 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.0703125 disk.dm-1.in‚ 0.0703125" disk.dm-0.out‚22233.2607421875 disk.sda1.out‚0 network.sent‚ 1.196563e+06 disk.nvme0n1.in‚ 334.48046875% disk.nvme0n1.out‚22233.2607421875 disk.drbd1777.in‚0 cpu‚25.049676564735897 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.rbd0.out‚0# memory_percent‚93.8319206237793' proc.memory.availableMB‚ 1010.578125 disk.rbd3.in‚0 disk.rbd0.in‚0 network.recv‚385817 disk.sda3.out‚ 674.41796875 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 proc.memory.rssMB‚ 2747.71875Ò‹“mô‚(:… Ë‚ãÔ¨ãëÓ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.357' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*D³@õ‚(:µ Ë‚ãÔ²ö™Ö disk.sda.in‚ 0.0703125 disk.sda1.in‚0 network.recv‚389596 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 disk.sda.out‚ 716.7890625 disk.rbd2.out‚53.95166015625 disk.sda2.in‚0$ memory_percent‚93.83206367492676) disk./.usagePercent‚72.60687305884848 disk.dm-1.in‚ 0.0703125 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 334.48046875 disk.sda3.out‚ 716.7890625) proc.memory.percent‚16.77074432373047 disk.dm-1.out‚ 716.7890625 disk.rbd0.in‚0# disk.dm-0.out‚22331.85302734375 disk.drbd1777.in‚0 disk.sda3.in‚ 0.0703125( proc.memory.availableMB‚ 1010.5546875 network.sent‚ 1.201793e+06 disk.drbd1777.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 334.48046875& disk.nvme0n1.out‚22331.85302734375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0% disk./.usageGB‚1081.8948783874512 disk.nvme0n1.in‚ 334.48046875 proc.memory.rssMB‚ 2747.71875 cpu‚25.049939374659996( disk.nvme0c0n1.out‚22331.85302734375Ì^Ëÿ0ö+ _wandbruntime‚920  _runtime‚920C™gÓÅ÷¿ ' lr-AdamW/pg1‚0.00016583095634644168 ' lr-AdamW/pg2‚0.00016583095634644168 trainer/global_step‚899 $ _timestamp‚1.788395857090927e+09 _runtime‚ 920.650905628 _step‚68Q·fÉøà _step‚68 ' lr-AdamW/pg1‚0.00016583095634644168 ' lr-AdamW/pg2‚0.00016583095634644168 trainer/global_step‚899 $ _timestamp‚1.788395857090927e+09 _runtime‚ 920.650905628D?ÿÇ0ù+ _wandbruntime‚920  _runtime‚920òÅ1Á'ú»O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1627317725184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1627550543872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33447487960576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23540617 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚69 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40702319379968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39914180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40702601234944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35074805685760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23515933 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 257914880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1648057519616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 920.651899261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6222266 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 trainer/global_step‚899 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16373563 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6222962 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23540617 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39889494 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16373563 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33447505963520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1627317725184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39054261860352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33072525720694 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35075056507392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39054285800448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1622430767222 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15821386 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33447487960576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23515933 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35075056507392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23540617 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚69 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40702319379968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39914180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 310063616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40702601234944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35074805685760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23515933 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 257914880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1648057519616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 920.651899261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6222266 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 trainer/global_step‚899 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16373563 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6222962 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23540617 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39889494 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16373563 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33447505963520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1627317725184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39054261860352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33072525720694 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35075056507392 dQDeviceStatsMonitor.on_train_bat‹ÿ.2ch_start/inactive_split_bytes.large_pool.allocated‚39054285800448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1622430767222 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15821386 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33447487960576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23515933 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35075056507392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39097640770048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6229820 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39933748 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6229098 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40747812170240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33484630250496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16391742 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1650147460096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33109256147718 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 % _timestamp‚1.7883958580592906e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15839595 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31484775645184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39958460 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9609775 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23542008 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35114007959040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23542008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9609773 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35114007959040 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16391740 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1629359705600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 _step‚70 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23566718 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33484648253440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1629359705600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33484648253440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39933748 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 _runtime‚ 921.620082488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31484792684544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15838871 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39097664710144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39958460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚899Ȉ‘8Ï&þÉM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚722 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39097640770048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6229820 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39933748 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6229098 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40747812170240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33484630250496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16391742 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1650147460096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33109256147718 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 % _timestamp‚1.7883958580592906e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15839595 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31484775645184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39958460 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9609775 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257901568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23542008 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35114007959040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 310063616 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23542008 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281841664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9609773 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35114007959040 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16391740 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1629359705600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 _step‚70 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23566718 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33484648253440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1629359705600 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33484648253440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39933748 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 _runtime‚ 921.620082488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31484792684544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15838871 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39097664710144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39958460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚899 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37022392016384 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35304602456576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6569894 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 279757824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35304602456576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10131772 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17282513 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10131770 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35304620459520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37022642838016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24847719 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1718022378496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37022392016384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1717789559808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34909060652702 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 trainer/global_step‚949 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16700974 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17282511 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1718022378496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684Cï+Æ'ÀO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17282511 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1718022378496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 255817728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚ 974.07100232 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24823035 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35304620459520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16701666 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41223231285248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24847719 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42962819220992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1712859992734 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17282513 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1712631030430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1717789559808 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37022392016384 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35304602456576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6569894 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 279757824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35304602456576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10131772 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17282513 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10131770 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35304620459520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37022642838016 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24847719 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1718022378496 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37022392016384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1717789559808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34909060652702 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 trainer/global_step‚949 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16700974 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686I!Ïb›Ž‚(: ‡ƒãÔÒŸÚÒ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.134' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.026799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sp2?‚(:´ ‡ƒãÔñÊœÕ proc.memory.rssMB‚ 2747.71875) proc.memory.percent‚16.77074432373047 disk.sda3.out‚ 772.05078125' disk.nvme0c0n1.out‚22530.3115234375 disk.rbd0.in‚0 network.sent‚ 1.268866e+06 disk.rbd2.out‚ 55.7060546875 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.dm-1.out‚ 772.05078125% disk.nvme0n1.out‚22530.3115234375 disk.sda.out‚ 772.05078125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda.in‚ 0.0703125$ memory_percent‚93.83490085601807) disk./.usagePercent‚72.60692528438545 disk.rbd1.out‚ 0.36962890625 disk.dm-1.in‚ 0.0703125 disk.rbd0.out‚0) proc.memory.availableMB‚ 1010.08984375 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0703125 cpu‚25.05049355537713 disk.sda1.in‚0 disk.drbd1777.out‚0 network.recv‚406895 disk.dm-0.in‚ 334.48046875 proc.cpu.threads‚17" disk.dm-0.out‚22530.3115234375% disk./.usageGB‚1081.8956565856934 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.48046875 disk.drbd1777.in‚0QS¥ò0+ _wandbruntime‚975  _runtime‚975#Ò…Ë&‘ÅM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37061594289664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17300692 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43008311412224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 % _timestamp‚1.7883959115047665e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10142423 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37061594289664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1719831540224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1719598708224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 286035968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34945545077854 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24873820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41266586254848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35341744746496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1741439121408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42149800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6575981 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34945791079726 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 _runtime‚ 975.0650828 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17300690 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1719598708224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 312159744 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1714663726382 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16718404 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43008025376256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 262095872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1714434763870 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17300692 _step‚74 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24849110 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35341762749440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24873820 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17300690 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33231110313984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚35341744746496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1741701217280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42174512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24849110 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33231127353344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42174512 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6576646 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 trainer/global_step‚949^­(z´’ÉM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24849110 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35341762749440 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2487382…'á’#0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17300690 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33231110313984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚35341744746496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1741701217280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42174512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24849110 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33231127353344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42174512 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6576646 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 trainer/global_step‚949 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37061594289664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17300692 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43008311412224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 % _timestamp‚1.7883959115047665e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10142423 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37061594289664 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1719831540224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1719598708224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 286035968 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34945545077854 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24873820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41266586254848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35341744746496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1741439121408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42149800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6575981 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34945791079726 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 _runtime‚ 975.0650828 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17300690 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1719598708224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 312159744 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1714663726382 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16718404 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43008025376256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 262095872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1714434763870 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17300692 _step‚74JþN.I0“+ _wandbruntime‚975  _runtime‚975¡’ÅGS”Í /loss/FRt4/assignment_loss‚5.498321104932715 2loss/SRqqt4/assignment_loss‚2.8524530094061324 0loss/SRqqt1/detection_loss‚3.733063181412631 .loss/FBt3/detection_loss‚3.974552293624466 %loss/total_loss‚81.44775778125546 /loss/FRt3/assignment_loss‚6.688367450938506 1loss/SRqqt1/assignment_loss‚3.753202167050592 .loss/FRt3/detection_loss‚2.784812962892009 1loss/SRqqt4/detection_loss‚1.2898046402972634 epoch‚0 trainer/global_step‚949 /loss/FRt1/assignment_loss‚7.084746190479824 1loss/FBt1/assignment_loss‚0.32264305153512396 1loss/SRqqt2/detection_loss‚1.6334784022470314 1loss/SRqqt3/assignment_loss‚3.190620709438713 1loss/SRqqt3/detection_loss‚2.7725555369301107 .loss/FBt4/detection_loss‚9.003764975070954 1loss/FBt3/assignment_loss‚0.28198760555419394 .loss/FBt2/detection_loss‚2.667438770087916 _step‚75 /loss/FBt2/assignment_loss‚0.832262920708779 /loss/FBt1/detection_loss‚5.1672173621399065 % _timestamp‚1.7883959115065205e+09 /loss/FRt1/detection_loss‚3.7515465702329363 /loss/FRt2/detection_loss‚2.7244907140731813 0loss/FRt2/assignment_loss‚5.2366838455200195 1loss/SRqqt2/assignment_loss‚3.133316993713379 1loss/FBt4/assignment_loss‚0.44486909456463763 _runtime‚ 975.066248607 /loss/FRt4/detection_loss‚2.6255582284044334 ïÃW•Ñ /loss/FRt4/assignment_loss‚5.498321104932715 2loss/SRqqt4/assignment_loss‚2.8524530094061324 0loss/SRqqt1/detection_loss‚3.733063181412631 .loss/FBt3/detection_loss‚3.974552293624466 %loss/total_loss‚81.44775778125546 /loss/FRt3/assignment_loss‚6.688367450938506 1loss/SRqqt1/assignment_loss‚3.753202167050592 .loss/FRt3/detection_loss‚2.784812962892009 1loss/SRqqt4/detection_loss‚1.2898046402972634 epoch‚0 trainer/global_step‚949 /loss/FRt1/assignment_loss‚7.084746190479824 1loss/FBt1/assignment_loss‚0.32264305153512396 1loss/SRqqt2/detection_loss‚1.6334784022470314 1loss/SRqqt3/assignment_loss‚3.190620709438713 1loss/SRqqt3/detection_loss‚2.7725555369301107 .loss/FBt4/detection_loss‚9.003764975070954 1loss/FBt3/assignment_loss‚0.28198760555419394 .loss/FBt2/detection_loss‚2.667438770087916 _step‚75 /loss/FBt2/assignment_loss‚0.832262920708779 /loss/FBt1/detection_loss‚5.1672173621399065 % _timestamp‚1.7883959115065205e+09 /loss/FRt1/detection_loss‚3.7515465702329363 /loss/FRt2/detection_loss‚2.7244907140731813 0loss/FRt2/assignment_loss‚5.2366838455200195 1loss/SRqqt2/assignment_loss‚3.133316993713379 1loss/FBt4/assignment_loss‚0.44486909456463763 _runtime‚ 975.066248607 /loss/FRt4/detection_loss‚2.6255582284044334KX`å=–‚(:² –ƒãÔŽæ¹Ö disk.dm-1.out‚ 790.21484375& disk.nvme0n1.out‚22628.86474609375 disk.sda.out‚ 790.21484375 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚22628.86474609375 disk.nvme0n1.in‚ 334.48046875& proc.memory.availableMB‚ 1010.28125 proc.cpu.threads‚17 disk.rbd2.out‚ 55.8310546875 disk.rbd1.out‚ 0.36962890625" disk.nvme0c0n1.in‚ 334.48046875 disk.dm-0.in‚ 334.48046875 disk.sda.in‚ 0.0703125) proc.memory.percent‚16.77074432373047 cpu‚25.05042346708921) disk./.usagePercent‚72.60690377975259 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.0703125# disk.dm-0.out‚22628.86474609375$ memory_percent‚93.83373260498047 proc.memory.rssMB‚ 2747.71875 disk.dm-1.in‚ 0.0703125 network.recv‚412277 disk.sda1.in‚0 disk.sda3.out‚ 790.21484375 network.sent‚ 1.30748e+06$ disk./.usageGB‚1081.895336151123 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0Wý…›—‚(: –ƒãÔÑ·Ö gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.657200000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ¸i˜‚(:… ¥ƒãÔæ™ÑÓ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.893' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7786 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù…c?™‚(:´ ¥ƒãÔ¼‚¢Ö" disk.dm-0.out‚22727.7919921875 network.recv‚416689 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 57.4208984375 disk.rbd1.out‚ 0.36962890625 disk.dm-1.in‚ 0.0703125 disk.dm-1.out‚ 825.42578125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 334.48046875% disk./.usageGB‚1081.8953971862793 disk.rbd1.in‚0 cpu‚25.05066931985897) disk./.usagePercent‚72.60690787587313 disk.sda.in‚ 0.0703125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 825.42578125" disk.nvme0c0n1.in‚ 334.48046875' disk.nvme0c0n1.out‚22727.7919921875$ memory_percent‚93.83375644683838 disk.sda.out‚ 825.42578125 disk.sda3.in‚ 0.0703125 disk.nvme0n1.in‚ 334.48046875% disk.nvme0n1.out‚22727.7919921875) proc.memory.availableMB‚ 1010.27734375 proc.memory.rssMB‚ 2747.71875 proc.cpu.threads‚17 disk.rbd0.in‚0) proc.memory.percent‚16.77074432373047 network.sent‚ 1.345969e+06 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0GØâßš‚(:… ´ƒãÔŸŽËÓ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚81.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.3048 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Ìs 6›‚(:« ´ƒãÔÅï’Ö) disk./.usagePercent‚72.60690659583547 disk.dm-1.out‚ 843.640625 disk.rbd0.out‚0" disk.dm-0.out‚22728.1396484375 disk.drbd1777.out‚0' proc.memory.availableMB‚ 1010.265625 disk.dm-1.in‚ 0.0703125 disk.sda1.out‚0) proc.memory.percent‚16.77074432373047 disk.rbd2.out‚ 57.5458984375 disk.sda2.in‚0 disk.sda3.out‚ 843.640625' disk.nvme0c0n1.out‚22728.1396484375 network.recv‚421139 proc.cpu.threads‚17 disk.sda.in‚ 0.0703125 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.36962890625 disk.rbd0.in‚0 disk.dm-0.in‚ 334.48046875 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2747.71875$ disk./.usageGB‚1081.895378112793 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.0703125 disk.nvme0n1.in‚ 334.48046875% disk.nvme0n1.out‚22728.1396484375$ memory_percent‚93.83382797241211 cpu‚25.05066502799372" disk.nvme0c0n1.in‚ 334.48046875 disk.sda1.in‚0 network.sent‚ 1.352269e+06 disk.rbd2.in‚0 disk.sda.out‚ 843.640625ªæ!j2œ- _wandbruntime‚1027  _runtime‚1027W4ŸÐÆÀ % _timestamp‚1.7883959636983697e+09 _runtime‚ 1027.25820108 _step‚76 ' lr-AdamW/pg1‚0.00018427711389332063 ' lr-AdamW/pg2‚0.00018427711389332063 trainer/global_step‚999_ÁúýÊžÄ trainer/global_step‚999 % _timestamp‚1.7883959636983697e+09 _runtime‚ 1027.25820108 _step‚76 ' lr-AdamW/pg1‚0.00018427711389332063 ' lr-AdamW/pg2‚0.00018427711389332063LÞ‘¸ 2Ÿ- _wandbruntime‚1027  _runtime‚1027 –¶Ã' ½O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6913919 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45223416922624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38969978347008 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37161716952576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10664422 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6914608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1831264092672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18191461 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1808494213120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17579030 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38969978347008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1808261394432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45223694583296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37161716952576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37161734955520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18191461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36745595584710 $ _timestamp‚1.788395963699176e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1808494213120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44346284 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26130137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1803060255942 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1802831293638 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1831517813248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34942535328768 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38970229168640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43392152829952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10664420 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26154821 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 _runtime‚1027.259273452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18191463 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26154821 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38970229168640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43392176770048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 253720576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44321598 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26130137 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1808494213120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44346284 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26130137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1803060255942 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1802831293638 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1831517813248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34942535328768 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38970229168640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43392152829952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10664420 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26154821 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 _runtime‚1027.259273452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18191463 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228962304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26154821 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38970229168640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43392176770048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 253720576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44321598 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26130137 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38969978347008 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37161716952576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10664422 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6914608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1831264092672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18191461 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1808494213120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17579030 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38969978347008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1808261394432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45223694583296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37161716952576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37161734955520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18191461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36745595584710 $ _timestamp‚1.788395963699176e+09M*jeÀ2¢- _wandbruntime‚1028  _runtime‚1028bµvÎ&£ÈM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1810070542848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6921425 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26156212 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37198877245440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288133120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚999 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37198877245440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26180922 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 312159744 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36782326011734 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26180922 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 % _timestamp‚1.7883959647506976e+09 _runtime‚1028.311393098 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34977462022144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37198859242496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 _step‚78 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1833353065472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39009180620288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1804863989590 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1833088872448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18209642 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6920734 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264193024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39009180620288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1810303374848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17596500 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18209642 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44365852 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26156212 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36782080009862 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44365852 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18209640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17595807 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44390564 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18209640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45268620612096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44390564 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10675073 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43435555679744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39008929785344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1810070542848 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45268908745216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34977444982784ûýÒ&¤ÌM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264193024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39009180620288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1810303374848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17596500 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18209642 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44365852 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26156212 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36782080009862 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44365852 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18209640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17595807 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44390564 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18209640 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45268620612096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44390564 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10675073 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43435555679744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39008929785344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1810070542848 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45268908745216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34977444982784 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1810070542848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6921425 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26156212 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37198877245440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288133120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚999 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37198877245440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26180922 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 312159744 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36782326011734 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26180922 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 % _timestamp‚1.7883959647506976e+09 _runtime‚1028.311393098 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34977462022144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37198859242496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 _step‚78 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1833353065472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39009180620288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1804863989590 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1833088872448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18209642 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6920734 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0NÑ«Ú2¥- _wandbruntime‚1028  _runtime‚1028ÿª¤6O¦É /loss/FRt3/assignment_loss‚6.763149914918123 0loss/SRqqt1/detection_loss‚3.776548203613077 0loss/SRqqt2/detection_loss‚1.753063339494953 .loss/FBt1/detection_loss‚2.834964493487744 /loss/FBt4/detection_loss‚3.9272852490345636 trainer/global_step‚999 2loss/SRqqt1/assignment_loss‚3.1024667535509383 .loss/FBt4/assignment_loss‚0.16865301034674 1loss/SRqqt3/detection_loss‚3.6393985774367867 1loss/FBt1/assignment_loss‚0.23248945842274127 0loss/FBt3/assignment_loss‚0.7975494908694398 .loss/FRt1/detection_loss‚3.382199501991272 1loss/SRqqt4/detection_loss‚1.4676993640052323 .loss/FBt3/detection_loss‚7.084711906207125 % _timestamp‚1.7883959647528186e+09 /loss/FRt2/assignment_loss‚5.348347514174705 2loss/SRqqt4/assignment_loss‚2.5963864854403904 .loss/FRt2/detection_loss‚3.380139944165252 .loss/FBt2/detection_loss‚4.802283764909953 epoch‚0 _runtime‚1028.312515273 /loss/FRt1/assignment_loss‚7.082607491811117 .loss/FRt3/detection_loss‚2.542536108168187 /loss/FRt4/detection_loss‚2.5470464853745587 _step‚79 %loss/total_loss‚79.58170338146456 2loss/SRqqt2/assignment_loss‚3.0671114701973763 0loss/FBt2/assignment_loss‚0.8376960158348083 /loss/FRt4/assignment_loss‚5.173921535599907 1loss/SRqqt3/assignment_loss‚3.273447302409581›Ô­S§Í 1loss/FBt1/assignment_loss‚0.23248945842274127 0loss/FBt3/assignment_loss‚0.7975494908694398 .loss/FRt1/detection_loss‚3.382199501991272 1loss/SRqqt4/detection_loss‚1.4676993640052323 .loss/FBt3/detection_loss‚7.084711906207125 % _timestamp‚1.7883959647528186e+09 /loss/FRt2/assignment_loss‚5.348347514174705 2loss/SRqqt4/assignment_loss‚2.5963864854403904 .loss/FRt2/detection_loss‚3.380139944165252 .loss/FBt2/detection_loss‚4.802283764909953 epoch‚0 _runtime‚1028.312515273 /loss/FRt1/assignment_loss‚7.082607491811117 .loss/FRt3/detection_loss‚2.542536108168187 /loss/FRt4/detection_loss‚2.5470464853745587 _step‚79 %loss/total_loss‚79.58170338146456 2loss/SRqqt2/assignment_loss‚3.0671114701973763 0loss/FBt2/assignment_loss‚0.8376960158348083 /loss/FRt4/assignment_loss‚5.173921535599907 1loss/SRqqt3/assignment_loss‚3.273447302409581 /loss/FRt3/assignment_loss‚6.763149914918123 0loss/SRqqt1/detection_loss‚3.776548203613077 0loss/SRqqt2/detection_loss‚1.753063339494953 .loss/FBt1/detection_loss‚2.834964493487744 /loss/FBt4/detection_loss‚3.9272852490345636 trainer/global_step‚999 2loss/SRqqt1/assignment_loss‚3.1024667535509383 .loss/FBt4/assignment_loss‚0.16865301034674 1loss/SRqqt3/detection_loss‚3.6393985774367867O §€d›¨‚(: ÃãÔήÈÔ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚76.108' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.221600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸æ­ØC©‚(:¸ ÃãÔ§ãŽÖ disk.nvme0n1.in‚ 334.48046875 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 cpu‚25.050892394475973 disk.rbd0.in‚0 disk.rbd2.out‚ 57.5576171875' proc.memory.availableMB‚ 1009.953125) proc.memory.percent‚16.77074432373047 disk.dm-0.in‚ 334.48046875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.36962890625 network.sent‚ 1.35755e+06 disk.sda3.in‚ 0.07421875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚72.60691171598614% disk./.usageGB‚1081.8954544067383 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 872.91796875& disk.nvme0n1.out‚22826.48974609375 disk.sda.out‚ 872.91796875 disk.drbd1777.out‚0# disk.dm-0.out‚22826.48974609375( disk.nvme0c0n1.out‚22826.48974609375$ memory_percent‚93.83573532104492 proc.memory.rssMB‚ 2747.71875 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 872.91796875" disk.nvme0c0n1.in‚ 334.48046875 network.recv‚424607ƒ]Aª‚(:… ÒƒãÔ¤‰þÒ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚77.996' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.5992 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ìúk=«‚(:² ÒƒãÔäïÄÕ$ disk./.usageGB‚1081.895378112793 disk.rbd0.in‚0 disk.dm-0.in‚ 334.48046875" disk.nvme0c0n1.in‚ 334.48046875# memory_percent‚93.8342809677124 disk.sda2.in‚0 disk.sda3.out‚ 889.546875 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875& disk.nvme0n1.out‚22826.82177734375# disk.dm-0.out‚22826.82177734375 disk.sda1.in‚0 disk.rbd2.in‚0 network.sent‚ 1.40756e+06 disk.nvme0n1.in‚ 334.48046875 disk.sda.out‚ 889.546875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd2.out‚57.64404296875( disk.nvme0c0n1.out‚22826.82177734375) proc.memory.availableMB‚ 1010.19140625) proc.memory.percent‚16.77074432373047 cpu‚25.05090801426267 proc.cpu.threads‚17 disk.dm-1.out‚ 889.546875 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 network.recv‚430353 disk.rbd3.out‚0 proc.memory.rssMB‚ 2747.71875) disk./.usagePercent‚72.60690659583547 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 0.36962890625t¢¤3›¬‚(: áƒãÔÚ¶Ó gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚77.839' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.567800000000002 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tõ4.6­‚(:« áƒãÔÜÿÒÖ disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 1010.4375 cpu‚25.05107113270259% disk./.usageGB‚1081.8955345153809 disk.nvme0n1.in‚ 334.48046875 disk.sda2.out‚0 disk.rbd1.out‚ 0.36962890625 disk.dm-1.in‚ 0.07421875 proc.memory.rssMB‚ 2747.71875 network.sent‚ 1.413129e+06# disk.dm-0.out‚22925.27490234375 proc.cpu.threads‚17& disk.nvme0n1.out‚22925.27490234375 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚72.60691709214436 disk.rbd2.out‚57.64404296875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚924.3125 disk.drbd1777.out‚0 disk.dm-0.in‚ 334.48046875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 334.48046875( disk.nvme0c0n1.out‚22925.27490234375$ memory_percent‚93.83277893066406 disk.dm-1.out‚924.3125) proc.memory.percent‚16.77074432373047 network.recv‚434890 disk.sda.in‚ 0.07421875 disk.sda.out‚924.3125Ö§½®‚(:„ ðƒãÔñœÓ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.045' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.609 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sûC#–¯‚(:· ðƒãÔû‰ŸÖ disk.drbd1777.in‚0$ memory_percent‚93.83432865142822 proc.cpu.threads‚17' disk./.usagePercent‚72.606910691956 disk.sda3.out‚ 926.1953125 disk.dm-0.in‚ 334.48046875 disk.sda.out‚ 926.1953125 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2747.71875 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0) proc.memory.percent‚16.77074432373047 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 926.1953125& disk.nvme0n1.out‚23023.68505859375# disk.dm-0.out‚23023.68505859375 disk.sda1.in‚0 disk.rbd3.in‚0 cpu‚25.051077236477216<¾äk¬ network.sent‚ 1.419421e+06 network.recv‚439518" disk.nvme0c0n1.in‚ 334.48046875 disk.rbd3.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚23023.68505859375 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0) proc.memory.availableMB‚ 1010.18359375 disk.rbd1.out‚ 0.36962890625 disk.rbd2.out‚57.72216796875% disk./.usageGB‚1081.8954391479492 disk.nvme0n1.in‚ 334.48046875 disk.sda1.out‚01PÂ52°- _wandbruntime‚1080  _runtime‚1080þSöDZÁ ' lr-AdamW/pg1‚0.00019350019266676008 ' lr-AdamW/pg2‚0.00019350019266676008 trainer/global_step‚1049 % _timestamp‚1.7883960173924081e+09 _runtime‚ 1080.95220754 _step‚80ïï¦:˲Š_runtime‚ 1080.95220754 _step‚80 ' lr-AdamW/pg1‚0.00019350019266676008 ' lr-AdamW/pg2‚0.00019350019266676008 trainer/global_step‚1049 % _timestamp‚1.7883960173924081e+09PE'[2³- _wandbruntime‚1080  _runtime‚1080}ñÄ'´¾O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40917564677632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40917564677632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19100413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40917815499264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18457447 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45561098314752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39018831448576 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19100411 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1898966047744 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39018831448576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27437239 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1898966047744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38582130516718 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40917815499264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1898733229056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19100411 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27437239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7260377 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 _runtime‚ 1080.95330036 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11197070 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36688852958208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11197072 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 trainer/global_step‚1049 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18458151 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27461923 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39018849451520 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40917564677632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40917564677632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19100413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40917815499264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18457447 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45561098314752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39018831448576 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19100411 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1898966047744 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39018831448576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27437239 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1898966047744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38582130516718 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40917815499264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1898733229056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19100411 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27437239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7260377 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 _runtime‚ 1080.95330036 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11197070 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36688852958208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11197072 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 trainer/global_step‚1049 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18458151 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27461923 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39018849451520 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46606616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1900542377472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19118590 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19118590 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚1081.977838749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39055991741440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27463314 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18474849 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39055991741440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1900775209472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7267126 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1894835290286 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45604501164544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1895064252798 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788396018416777e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39055973738496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39055973738496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27463314 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40956766950912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47529434112000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1924932947456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27488024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27488024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1900775209472 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40956766950912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1900542377472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19118592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11207723 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46581904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46581904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47529150173184 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38618614941870 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36723779651584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19118592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559Çæñ Œ¸ÌM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27488024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27488024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1900775209472 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40956766950912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1900542377472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19118592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1+9ûF!1207723 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46581904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46581904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47529150173184 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38618614941870 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36723779651584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19118592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40956516115968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18475519 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283938816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45604477224448 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46606616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1900542377472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19118590 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19118590 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚1081.977838749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39055991741440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27463314 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18474849 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39055991741440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1900775209472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7267126 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1894835290286 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45604501164544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1895064252798 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788396018416777e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39055973738496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39055973738496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27463314 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40956766950912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚47529434112000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1924932947456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42865151008256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1989437882368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40875963947520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20009361 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2014420818432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11729720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788396070445247e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40418419447062 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40418665448726 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1989205063680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48778388 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11729722 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49744744375808 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28769025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1983460782358 trainer/global_step‚1099 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42865401829888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40875963947520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7608020 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38435204666368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20009361 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19337053 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1989437882368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47730067739648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 255817728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1983231820054 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42865151008256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42865151008256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1989437882368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40875963947520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20009361 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2014420818432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11729720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788396070445247e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40418419447062 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40418665448726 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1989205063680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250821632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48778388 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11729722 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49744744375808 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28769025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1983460782358 trainer/global_step‚1099 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42865401829888 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40875963947520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7608020 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38435204666368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20009361 M@DeviceStatsMonitor.on_train_batch_start/inactive_splÑÁtÊ€it.all.freed‚19337053 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1989437882368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47730067739648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 255817728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1983231820054 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42865151008256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48822668 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48797956 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49789958597632 trainer/global_step‚1099 _step‚86 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2016511948288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38470114320384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28795126 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1991247044096 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49789670464512 _runtime‚1134.992522137 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28770416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1991247044096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19354287 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7613914 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28770416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40455149873878 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1991014212096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42904353281536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264193024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47773422709248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288133120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20027542 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48822668 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 312159744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40455395875750 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872äÔJwÓ&ÊÍM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 288133120 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20027542 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48822668 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 312159744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40455395875750 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48822668 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48797956 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49789958597632 trainer/global_step‚1099 _step‚86 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2016511948288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38470114320384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28795126 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1991247044096 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49789670464512 _runtime‚1134.992522137 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28770416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1991247044096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19354287 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7613914 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28770416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40455149873878 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1991014212096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42904353281536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 264193024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47773422709248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0VC A2Ë- _wandbruntime‚1134  _runtime‚1134pª=OÌÉ .loss/FRt4/detection_loss‚2.279654438125676 1loss/SRqqt1/detection_loss‚3.3199377014086795 _runtime‚1134.993675371 0loss/FBt2/assignment_loss‚0.8318557143211365 .loss/SRqqt2/detection_loss‚1.4281045831498 %loss/total_loss‚71.63448110958683 _step‚87 /loss/FRt1/assignment_loss‚6.063592700958252 /loss/FRt2/assignment_loss‚5.545931873650386 2loss/SRqqt1/assignment_loss‚2.8727831978064318 1loss/FBt1/assignment_loss‚0.26670377940971257 1loss/SRqqt3/detection_loss‚2.9391340880821915 1loss/FBt3/assignment_loss‚0.41716835900608984 0loss/FBt4/assignment_loss‚0.2658995759717072 -loss/FBt2/detection_loss‚4.48041220144792 trainer/global_step‚1099 /loss/FRt3/assignment_loss‚6.618507080889763 2loss/SRqqt3/assignment_loss‚3.2989580753522043 1loss/SRqqt4/assignment_loss‚2.456453052567847 .loss/FRt2/detection_loss‚2.336521655834954 /loss/FRt3/detection_loss‚2.8203720241150956 .loss/FBt4/detection_loss‚4.233943837881088 .loss/FRt1/detection_loss‚2.954629645347595 .loss/FBt1/detection_loss‚2.470001485417871 1loss/SRqqt4/detection_loss‚1.2372878584737694 .loss/FBt3/detection_loss‚4.970276816260247 epoch‚0 /loss/FRt4/assignment_loss‚4.679937609310808 1loss/SRqqt2/assignment_loss‚2.846413754797601 % _timestamp‚1.7883960714339619e+09ê4qSÍÍ 0loss/FBt2/assignment_loss‚0.8318557143211365 .loss/SRqqt2/detection_loss‚1.4281045831498 %loss/total_loss‚71.63448110958683 _step‚87 /loss/FRt1/assignment_loss‚6.063592700958252 /loss/FRt2/assignment_loss‚5.545931873650386 2loss/SRqqt1/assignment_loss‚2.8727831978064318 1loss/FBt1/assignment_loss‚0.26670377940971257 1loss/SRqqt3/detection_loss‚2.9391340880821915 1loss/FBt3/assignment_loss‚0.41716835900608984 0loss/FBt4/assignment_loss‚0.2658995759717072 -loss/FBt2/detection_loss‚4.48041220144792 trainer/global_step‚1099 /loss/FRt3/assignment_loss‚6.618507080889763 2loss/SRqqt3/assignment_loss‚3.2989580753522043 1loss/SRqqt4/assignment_loss‚2.456453052567847 .loss/FRt2/detection_loss‚2.336521655834954 /loss/FRt3/detection_loss‚2.8203720241150956 .loss/FBt4/detection_loss‚4.233943837881088 .loss/FRt1/detection_loss‚2.954629645347595 .loss/FBt1/detection_loss‚2.470001485417871 1loss/SRqqt4/detection_loss‚1.2372878584737694 .loss/FBt3/detection_loss‚4.970276816260247 epoch‚0 /loss/FRt4/assignment_loss‚4.679937609310808 1loss/SRqqt2/assignment_loss‚2.846413754797601 % _timestamp‚1.7883960714339619e+09 .loss/FRt4/detection_loss‚2.279654438125676 1loss/SRqqt1/detection_loss‚3.3199377014086795 _runtime‚1134.993675371W8Blç΂(:… ¬„ãÔçòäÒ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5874 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xàü¨3Ï‚(:¨ ¬„ãÔèà›Õ# disk.nvme0c0n1.out‚ 23221.515625 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 981.1796875 disk.rbd0.out‚0$ memory_percent‚93.83535385131836) proc.memory.percent‚16.77074432373047 network.recv‚457763 disk.rbd3.out‚0 disk.dm-0.out‚ 23221.515625 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 334.48046875 cpu‚25.051495326287924 disk.sda2.out‚0 disk.sda3.out‚ 981.1796875" disk.nvme0c0n1.in‚ 334.48046875% disk./.usageGB‚1081.8956260681152 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 981.1796875 disk.sda3.in‚ 0.07421875 proc.memory.rssMB‚ 2747.71875) disk./.usagePercent‚72.60692323632519 disk.rbd2.in‚0 disk.rbd2.out‚59.48486328125 disk.rbd1.out‚ 0.51806640625! disk.nvme0n1.out‚ 23221.515625 disk.sda.in‚ 0.07421875' proc.memory.availableMB‚ 1010.015625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.48046875 network.sent‚ 1.486686e+066úI•›Ð‚(: »„ãÔŽ§ýÒ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.531' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.906200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ( É@Ñ‚(:µ »„ãÔÍäÏÕ" disk.dm-0.out‚23320.1455078125 disk.sda1.out‚0 proc.memory.rssMB‚ 2747.71875 proc.cpu.threads‚17 disk.sda2.in‚0% disk.nvme0n1.out‚23320.1455078125 network.sent‚ 1.535539e+06 disk.drbd1777.out‚0 disk.sda3.out‚ 998.9453125" disk.nvme0c0n1.in‚ 334.48046875 disk.dm-1.out‚ 998.9453125 disk.nvme0n1.in‚ 334.48046875 network.recv‚462656 disk.rbd2.in‚0 disk.rbd2.out‚59.48486328125 disk.rbd1.out‚ 0.51806640625) proc.memory.percent‚16.77074432373047% disk./.usageGB‚1081.8956260681152 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 998.9453125 disk.sda1.in‚0$ memory_percent‚93.83540153503418 disk.sda3.in‚ 0.07421875' disk.nvme0c0n1.out‚23320.1455078125 disk.dm-0.in‚ 334.48046875 disk.sda.in‚ 0.07421875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚ 0.07421875) disk./.usagePercent‚72.60692323632519 disk.sda2.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 1010.0078125 cpu‚25.051458425302837lÃþý›Ò‚(: Ê„ãÔ™’äÒ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚84.278' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚16.855600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u‰íx7Ó‚(:¬ Ê„ãÔåâÉÕ" disk.dm-0.out‚23418.4111328125 disk.sda1.out‚0 disk.sda3.in‚ 0.07421875 network.recv‚466143 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 0.55029296875 proc.memory.rssMB‚ 2747.71875 disk.drbd1777.in‚0 disk.sda2.in‚0! memory_percent‚93.83544921875 cpu‚25.051442186905987 disk.nvme0n1.in‚ 334.48046875 disk.sda.in‚ 0.07421875 disk.rbd2.out‚61.07470703125 disk.sda3.out‚ 1016.7734375 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.54091e+06) disk./.usagePercent‚72.60692323632519 disk.sda1.in‚0 disk.sda2.out‚0% disk./.usageGB‚1081.8956260681152" disk.nvme0c0n1.in‚ 334.48046875' disk.nvme0c0n1.out‚23418.4111328125) proc.memory.percent‚16.77074432373047 disk.dm-1.out‚ 1016.7734375 disk.rbd0.out‚0% disk.nvme0n1.out‚23418.4111328125 disk.dm-0.in‚ 334.48046875 disk.sda.out‚ 1016.7734375 disk.rbd3.out‚0 proc.memory.availableMB‚1010 disk.rbd2.in‚0ðx ›Ô‚(: Ù„ãÔžÐúÒ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.005' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.400999999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÏÌ?Õ‚(:´ Ù„ãÔ’ïÅÖ disk.rbd0.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1010.25390625& disk.nvme0c0n1.out‚23419.244140625 disk.rbd3.out‚0 disk.sda2.in‚0) proc.memory.percent‚16.77074432373047 cpu‚25.05158921990218) disk./.usagePercent‚72.60686512261492" disk.nvme0c0n1.in‚ 334.48046875 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 network.sent‚ 1.546381e+06 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 disk.sda2.out‚0$ memory_percent‚93.83389949798584 proc.memory.rssMB‚ 2747.71875 proc.cpu.threads‚17 disk.sda.out‚ 1049.3984375 disk.rbd2.out‚61.07470703125 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 0.55029296875 disk.dm-1.out‚ 1049.3984375 disk.dm-0.in‚ 334.48046875! disk.dm-0.out‚23419.244140625 disk.sda1.out‚0 disk.drbd1777.out‚0$ disk./.usageGB‚1081.894760131836$ disk.nvme0n1.out‚23419.244140625 disk.sda1.in‚0 disk.sda3.out‚ 1049.3984375 network.recv‚470002 disk.nvme0n1.in‚ 334.48046875 disk.drbd1777.in‚0 disk.rbd3.in‚0K§{eÖ- _wandbruntime‚118ÎG÷¥6  _runtime‚1186ãç|øÈ× % _timestamp‚1.7883961233785825e+09 _runtime‚1186.938385707 _step‚88 ' lr-AdamW/pg1‚0.00021194635021363903 ' lr-AdamW/pg2‚0.00021194635021363903 trainer/global_step‚1149_ ÖAÌØÆ ' lr-AdamW/pg2‚0.00021194635021363903 trainer/global_step‚1149 % _timestamp‚1.7883961233785825e+09 _runtime‚1186.938385707 _step‚88 ' lr-AdamW/pg1‚0.00021194635021363903X§áµ…2Ù- _wandbruntime‚1186  _runtime‚1186YÑÇÅ'Ú¿O TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30076127 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517497301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20214495 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42255200380734 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651346893 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2079909716992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52005004786176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12262370 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20918313 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20918313 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44812988160512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42733078443520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚40181522295808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283952128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2079676898304 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7883961233794043e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42733060440576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 _runtime‚1186.939804191 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50969754 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50994440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2106275513856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49899013224448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42733078443520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42733060440576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚717 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2073661045566 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20213778 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42254954379070 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30076127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44812737338880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 260012032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42733060440576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 521397248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 _runtime‚1186.939804191 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24686 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50969754 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50994440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2106275513856 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49899013224448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42733078443520 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42733060440576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚717 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2073661045566 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20213778 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42254954379070 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30076127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44812737338880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 260012032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20918313 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44812988160512 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42733078443520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚40181522295808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6873438720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283952128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7033847808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2079676898304 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Yës~H2Ü- _wandbruntime‚1187  _runtime‚1187]„ø\Ï&ÝÉM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44851939612160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44851688777216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2081718878720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 trainer/global_step‚1149 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49942392134144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42291930807758 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51014008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20936492 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2107843634176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30077518 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2081718878720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52050211828224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20231376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42770220733440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚40216448989184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20936490 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42770220733440 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44851939612160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 _runtime‚1187.952877614 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30077518 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51014008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20936492 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40216466028544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20232086 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 % _timestamp‚1.7883961243917868e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30102228 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7959061 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30102228 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42770202730496 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52050493669888 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20936490 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2108101535744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51038720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42291684805886 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2075464779214 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2075235816702 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12273023 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49942368194048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12273025 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51038720S±‹DXÞÍM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51014008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20936492 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2107843634176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30077518 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2081718878720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52050211828224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20231376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42770220733440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retriá! {es‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚40216448989184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20936490 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42770220733440 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44851939612160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6873438720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 _runtime‚1187.952877614 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30077518 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51014008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20936492 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40216466028544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20232086 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268675584 % _timestamp‚1.7883961243917868e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30102228 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7959061 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30102228 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42770202730496 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52050493669888 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20936490 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2108101535744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 521397248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51038720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42291684805886 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2075464779214 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2075235816702 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12273023 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49942368194048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12273025 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51038720 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44851939612160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44851688777216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2081718878720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232832000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 trainer/global_step‚1149 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49942392134144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42291930807758 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ZŸç2ß- _wandbruntime‚1187  _runtime‚1187÷Âþ VàÐ 0loss/SRqqt3/detection_loss‚3.402802810072899 /loss/FBt1/detection_loss‚2.0627573614188455 epoch‚0 _step‚91 0loss/FRt3/assignment_loss‚6.2637608757725465 1loss/SRqqt1/assignment_loss‚3.145435703428168 2loss/SRqqt4/assignment_loss‚2.6259676733532467 2loss/SRqqt3/assignment_loss‚2.7839224423680986 0loss/FBt2/assignment_loss‚0.5043199170168544 .loss/FBt3/detection_loss‚8.285561577840285 trainer/global_step‚1149 /loss/FRt2/assignment_loss‚5.325824878833912 2loss/SRqqt2/assignment_loss‚2.7731008008122444 /loss/FRt4/detection_loss‚2.7375628628901074 .loss/FBt2/detection_loss‚5.091523153994999 % _timestamp‚1.7883961243943121e+09 /loss/FRt1/assignment_loss‚6.220190540436776 /loss/FRt3/detection_loss‚2.3710469239977776 /loss/SRqqt1/detection_loss‚3.82803150070341 _runtime‚1187.954093276 /loss/FRt4/assignment_loss‚4.236116920198713 0loss/FBt3/assignment_loss‚0.2306022706040742 1loss/SRqqt4/detection_loss‚1.4295194408466183 1loss/FBt1/assignment_loss‚0.37679750455049144 /loss/FBt4/detection_loss‚3.7081633399833334 %loss/total_loss‚74.30074798986355 .loss/FRt2/detection_loss‚2.421137246268767 1loss/SRqqt2/detection_loss‚1.6332466979045421 1loss/FBt4/assignment_loss‚0.20517529067695828 /loss/FRt1/detection_loss‚2.6381802558898926×NÔÅZáÔ /loss/FRt4/assignment_loss‚4.236116920198713 0loss/FBt3/assignment_loss‚0.2306022706040742 1loss/SRqqt4/detection_loss‚1.4295194408466183 1loss/FBt1/assignment_loss‚0.37679750455049144 /loss/FBt4/detection_loss‚3.7081633399833334 %loss/total_loss‚74.30074798986355 .loss/FRt2/detection_loss‚2.421137246268767 1loss/SRqqt2/detection_loss‚1.6332466979045421 1loss/FBt4/assignment_loss‚0.20517529067695828 /loss/FRt1/detection_loss‚2.6381802558898926 0loss/SRqqt3/detection_loss‚3.402802810072899 /loss/FBt1/detection_loss‚2.0627573614188455 epoch‚0 _step‚91 0loss/FRt3/assignment_loss‚6.2637608757725465 1loss/SRqqt1/assignment_loss‚3.145435703428168 2loss/SRqqt4/assignment_loss‚2.6259676733532467 2loss/SRqqt3/assignment_loss‚2.7839224423680986 0loss/FBt2/assignment_loss‚0.5043199170168544 .loss/FBt3/detection_loss‚8.285561577840285 trainer/global_step‚1149 /loss/FRt2/assignment_loss‚5.325824878833912 2loss/SRqqt2/assignment_loss‚2.7731008008122444 /loss/FRt4/detection_loss‚2.7375628628901074 .loss/FBt2/detection_loss‚5.091523153994999 % _timestamp‚1.7883961243943121e+09 /loss/FRt1/assignment_loss‚6.220190540436776 /loss/FRt3/detection_loss‚2.3710469239977776 /loss/SRqqt1/detection_loss‚3.82803150070341 _runtime‚1187.954093276[ºdTâ‚(:… è„ãÔÝ™åÓ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.357' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6714 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÕjÕ<ã‚(:± è„ãÔ€ùÕ" disk.nvme0c0n1.in‚ 334.48046875 disk.sda1.out‚0) disk./.usagePercent‚72.60686512261492$ disk./.usageGB‚1081.894760131836 disk.rbd2.out‚61.24755859375' proc.memory.availableMB‚ 1009.921875 proc.memory.rssMB‚ 2747.71875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 334.48046875! disk.dm-0.out‚23517.556640625 disk.rbd2.in‚0) proc.memory.percent‚16.77074432373047 network.recv‚474832 disk.sda.out‚ 1050.23828125& disk.nvme0c0n1.out‚23517.556640625 disk.dm-0.in‚ 334.48046875 disk.sda.in‚ 0.07421875 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875# memory_percent‚93.8359260559082 disk.rbd1.in‚0 network.sent‚ 1.552717e+06$ disk.nvme0n1.out‚23517.556640625 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 1050.23828125 cpu‚25.05159497120055 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.61328125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.07421875 disk.sda3.out‚ 1050.23828125]4ÝŒä‚(:… ÷„ãÔ¦ðŒÓ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚77.996' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.5992 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!²1Aå‚(:¶ ÷„ãÔóŸÖ disk.rbd0.out‚0 disk.rbd3.out‚0$ memory_percent‚93.83771419525146) proc.memory.availableMB‚ 1009.62890625# proc.memory.rssMB‚ 2747.72265625 disk.sda3.in‚ 0.07421875) proc.memory.percent‚16.77076816558838 disk.dm-1.out‚ 1081.87109375 disk.rbd2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 1081.87109375 disk.rbd1.out‚ 0.61328125 disk.nvme0n1.in‚ 334.48046875 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.sda2.in‚0 cpu‚25.051770480755525 network.sent‚ 1.601724e+06 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0# disk.nvme0n1.out‚23517.99609375 disk.rbd2.out‚61.24755859375 network.recv‚479605) disk./.usagePercent‚72.60685206623066 disk.sda.out‚ 1081.87109375 disk.drbd1777.out‚0% disk./.usageGB‚1081.8945655822754" disk.nvme0c0n1.in‚ 334.48046875 disk.dm-0.in‚ 334.48046875 disk.sda1.out‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚23517.99609375 disk.dm-0.out‚23517.99609375 disk.sda1.in‚0ç6þ›æ‚(: †…ãÔƯÅÔ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚85.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.164199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š-Q˜>ç‚(:³ †…ãÔ°§Ö$ disk./.usageGB‚1081.894775390625 disk.rbd3.in‚0 network.sent‚ 1.608019e+06 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚23616.607421875 proc.cpu.threads‚17 network.recv‚484057 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 1117.140625) disk./.usagePercent‚72.60686614664505 disk.sda1.in‚0 disk.dm-0.in‚ 334.48046875! disk.dm-0.out‚23616.607421875 disk.rbd1.out‚ 0.67626953125 disk.dm-1.out‚ 1117.140625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 334.48046875$ memory_percent‚93.83776187896729) proc.memory.availableMB‚ 1009.62109375 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875) proc.memory.percent‚16.77079200744629 disk.sda3.out‚ 1117.140625" disk.nvme0c0n1.in‚ 334.48046875" proc.memory.rssMB‚ 2747.7265625 cpu‚25.05194798881138 disk.rbd2.out‚61.32568359375 disk.rbd1.in‚0$ disk.nvme0n1.out‚23616.607421875 disk.sda.in‚ 0.07421875…€Z2è- _wandbruntime‚1239  _runtime‚1239¯²iÂÆéÀ % _timestamp‚1.7883961763139648e+09 _runtime‚1239.873918689 _step‚92 & lr-AdamW/pg1‚0.0002211694289870785 & lr-AdamW/pg2‚0.0002211694289870785 trainer/global_step‚1199;X—´ÊêÄ & lr-AdamW/pg1‚0.0002211694289870785 & lr-AdamW/pg2‚0.0002211694289870785 trainer/global_step‚1199 % _timestamp‚1.7883961763139648e+09 _runtime‚1239.873918689 _step‚92\ñ÷ÃX2ë- _wandbruntime‚1239  _runtime‚1239ZlŠáÄ'ì¾O trainer/global_step‚1199 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2170148732928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2170381551616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21094672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46760574491136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8299652 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281854976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2197866375680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 257914880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44590192939520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44590174936576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚737 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8300389 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53185806 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46760323669504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44091489311078 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21827261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53210492 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2163861308774 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31383229 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54265825084928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52067934769152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21827263 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41927874003968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31358545 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12795020 $ _timestamp‚1.788396176314845e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46760323669504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44091735312742 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21095411 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41927856964608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46760574491136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21827263 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2170148732928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307ÀÅ©íÂO trainer/global_step‚1199 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2170148732928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2170381551616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21094672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46760574491136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8299652 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281854976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2197866375680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 257914880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268675584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44590192939520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44590174936576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30535 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚737 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8300389 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53185806 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46760323669504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44091489311078 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21827261 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250821632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232818688 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53210492 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2163861308774 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31383229 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54265825084928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52067934769152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246001664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 312159744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232818688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21827263 GADeviceStatsMonitor.on(ø¯Ã _train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41927874003968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31358545 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30310 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12795020 $ _timestamp‚1.788396176314845e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46760323669504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 521397248 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44091735312742 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30310 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6873438720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21095411 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41927856964608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24686 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30535 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46760574491136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21827263 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2170148732928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307]¢+® 2î- _wandbruntime‚1240  _runtime‚1240·V`ÏÏ&ïÉM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53254772 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21845440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21112872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21845440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31409330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2199445356544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12805675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46799525942784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 277647360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52111337618944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1240.884183896 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46799275107840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 _step‚94 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 312159744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12805673 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165436079910 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44627317226496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44627335229440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44128219737894 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21845442 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41962783657984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8307197 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 253707264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53230060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44627335229440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2171957881344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44128465739766 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21112203 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44627317226496 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2172190713344UXËìÓ&ðÍM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12805673 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30535 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24710 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165436079910 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44627317226496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44627335229440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24710 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44128219737894 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 521397248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚559 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21845442 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41962783657984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8307197 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 253707264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6873438720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53230060 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44627335229440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30310 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2171957881344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44128465739766 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21112203 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44627317226496 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2172190713344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53254772 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21845440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21112872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7033847808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30535 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651346893 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21845440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31409330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250834944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2199445356544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12805675 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250834944 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46799525942784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 277647360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52111337618944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517497301 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232832000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1240.884183896 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46799275107840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246001872 _step‚94 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 312159744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228962512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559^:[œ2ñ- _wandbruntime‚1240  _runtime‚1240¦ÙvRòÌ /loss/FRt3/assignment_loss‚6.381732873916626 1loss/SRqqt4/assignment_loss‚2.582971064012442 .loss/FRt1/detection_loss‚2.612104361707514 /loss/FBt1/detection_loss‚2.4011819874169307 0loss/FBt3/assignment_loss‚0.5475827864632336 .loss/FRt3/detection_loss‚2.581674932911992 1loss/SRqqt3/detection_loss‚2.9095466606544726 _runtime‚1240.885460543 1loss/FBt4/assignment_loss‚0.23201719309533722 1loss/SRqqt2/detection_loss‚1.5877701617681828 /loss/FBt2/detection_loss‚2.4987728759188657 -loss/FBt3/detection_loss‚5.75620959355281 /loss/FRt4/assignment_loss‚3.812567384346672 1loss/SRqqt1/assignment_loss‚2.866367220878601 0loss/FBt1/assignment_loss‚0.4592502241070906 .loss/FRt2/detection_loss‚2.225764468067982 .loss/FRt4/detection_loss‚2.439677260492159 0loss/SRqqt1/detection_loss‚4.506210044026375 1loss/SRqqt4/detection_loss‚1.3597807077727448 0loss/FBt2/assignment_loss‚0.4654168427361583 trainer/global_step‚1199 _step‚95 /loss/FRt2/assignment_loss‚4.661710891567293 2loss/SRqqt3/assignment_loss‚2.7642326138236304 % _timestamp‚1.7883961773257225e+09 2loss/SRqqt2/assignment_loss‚2.5372712795551005 /loss/FBt4/detection_loss‚2.4962702321283743 %loss/total_loss‚67.23898039897296 epoch‚0 /loss/FRt1/assignment_loss‚6.552896738052368jEEVóÐ %loss/total_loss‚67.23898039897296 epoch‚0 /loss/FRt1/assignment_loss‚6.552896738052368 /loss/FRt3/assignment_loss‚6.381732873916626 1loss/SRqqt4/assignment_loss‚2.582971064012442 .loss/FRt1/detection_loss‚2.612104361707514 /loss/FBt1/detection_loss‚2.4011819874169307 0loss/FBt3/assignment_loss‚0.5475827864632336 .loss/FRt3/detection_loss‚2.581674932911992 1loss/SRqqt3/detection_loss‚2.9095466606544726 _runtime‚1240.885460543 1loss/FBt4/assignment_loss‚0.23201719309533722 1loss/SRqqt2/detection_loss‚1.5877701617681828 /loss/FBt2/detection_loss‚2.4987728759188657 -loss/FBt3/detection_loss‚5.75620959355281 /loss/FRt4/assignment_loss‚3.812567384346672 1loss/SRqqt1/assignment_loss‚2.866367220878601 0loss/FBt1/assignment_loss‚0.4592502241070906 .loss/FRt2/detection_loss‚2.225764468067982 .loss/FRt4/detection_loss‚2.439677260492159 0loss/SRqqt1/detection_loss‚4.506210044026375 1loss/SRqqt4/detection_loss‚1.3597807077727448 0loss/FBt2/assignment_loss‚0.4654168427361583 trainer/global_step‚1199 _step‚95 /loss/FRt2/assignment_loss‚4.661710891567293 2loss/SRqqt3/assignment_loss‚2.7642326138236304 % _timestamp‚1.7883961773257225e+09 2loss/SRqqt2/assignment_loss‚2.5372712795551005 /loss/FBt4/detection_loss‚2.4962702321283743_ Ñêô‚(:… •…ãÔ¼ûÒ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7426 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÍÃëHõ‚(:½ •…ãÔôÛÖ) disk./.usagePercent‚72.60693501267176% disk.nvme0n1.out‚23715.3349609375" disk.dm-0.out‚23715.3349609375 disk.rbd3.in‚0" proc.memory.rssMB‚ 2747.7265625 network.recv‚488178 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67626953125 disk.sda.in‚ 0.07421875' disk.nvme0c0n1.out‚23715.3349609375 disk.dm-1.in‚ 0.07421875# memory_percent‚93.8380479812622) proc.memory.availableMB‚ 1009.57421875 cpu‚25.051732866074563 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 334.48046875 disk.nvme0n1.in‚ 334.48046875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚61.32568359375 disk.dm-1.out‚ 1150.34765625 network.sent‚ 1.613581e+06% disk./.usageGB‚1081.8958015441895 disk.sda.out‚ 1150.34765625 disk.sda2.in‚0) proc.memory.percent‚16.77079200744629 disk.rbd0.out‚0 disk.dm-0.in‚ 334.48046875 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 1150.34765625ž ‘ö‚(:† ¤…ãÔ…σÔ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚117.142' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸˜p>÷‚(:³ ¤…ãÔˆÃÕ) disk./.usagePercent‚72.60693117255875 disk.rbd2.out‚62.97802734375 disk.rbd3.in‚0 disk.rbd1.out‚ 0.67626953125 disk.dm-1.out‚ 1151.09375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0" disk.dm-0.out‚23715.6708984375 network.recv‚493924% disk.nvme0n1.out‚23715.6708984375 disk.sda.out‚ 1151.09375) proc.memory.percent‚17.31736660003662% disk./.usageGB‚1081.8957443237305 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚23715.6708984375( proc.memory.availableMB‚ 919.73046875 cpu‚25.08185117055127 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 334.48046875# memory_percent‚94.3864107131958# proc.memory.rssMB‚ 2837.27734375 disk.sda2.out‚0 disk.sda3.out‚ 1151.09375 disk.nvme0n1.in‚ 334.48046875 disk.sda.in‚ 0.07421875 network.sent‚ 1.663555e+06p:D&šø‚(: ³…ãÔ¤ù¤Ó gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚83.742' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚16.748400000000004 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¡éÃDù‚(:¹ ³…ãÔÄ‘®Ö disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 network.sent‚ 1.668869e+06 disk.rbd3.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.nvme0n1.in‚ 334.48046875 disk.sda.in‚ 0.07421875* proc.memory.percent‚17.336225509643555 disk.sda3.out‚ 1182.3046875 disk.dm-1.out‚ 1182.3046875& disk.nvme0n1.out‚23814.50732421875 disk.sda.out‚ 1182.3046875 disk.sda1.in‚0 network.recv‚497599 disk.sda2.in‚0 disk.rbd1.out‚ 0.67626953125 disk.dm-0.in‚ 334.48046875 disk.drbd1777.in‚0( disk.nvme0c0n1.out‚23814.50732421875 disk.sda1.out‚0" proc.memory.rssMB‚ 2840.3671875% disk./.usageGB‚1081.8957633972168 disk.drbd1777.out‚0 disk.rbd2.out‚62.97802734375 disk.rbd3.in‚0# disk.dm-0.out‚23814.50732421875$ memory_percent‚94.40498352050781$ proc.memory.availableMB‚916.6875 cpu‚25.13081286217716) disk./.usagePercent‚72.60693245259642 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875&5î³ú‚(:… Â…ãÔüÍËÔ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚82.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.4104 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Dæ;û‚(:° Â…ãÔ矄× proc.cpu.threads‚17# disk.nvme0n1.out‚23916.17578125 disk.rbd3.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 334.48046875 disk.dm-0.in‚ 334.48046875 disk.sda.out‚ 1215.8046875 disk.rbd1.in‚0 disk.drbd1777.in‚0% disk./.usageGB‚1081.8988189697266 disk.rbd2.out‚ 63.142578125 disk.sda2.in‚0 disk.rbd1.out‚ 0.67626953125% disk.nvme0c0n1.out‚23916.17578125 disk.dm-1.out‚ 1215.8046875 disk.sda.in‚ 0.07421875 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 334.48046875' proc.memory.availableMB‚ 933.3671875! proc.memory.rssMB‚ 2823.703125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-0.out‚23916.17578125 disk.drbd1777.out‚0 network.sent‚ 1.677229e+06 network.recv‚502577) disk./.usagePercent‚72.60713751463135 disk.rbd2.in‚0 disk.sda3.out‚ 1215.8046875 disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚25.17904581093534 disk.rbd0.in‚0$ memory_percent‚94.30317878723145* proc.memory.percent‚17.234516143798828›ÏLŸ‘ü‚(:† Ñ…ãÔÂÖèÓ gpu.0.gpu‚45.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘˜7ý‚(:¾ Ñ…ãÔà¾ÂÖ disk.sda.out‚ 1216.480468b¼75 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.67626953125 proc.cpu.threads‚17 network.sent‚ 1.680646e+06* proc.memory.percent‚16.770386695861816 network.recv‚506602) disk./.usagePercent‚72.60711985011147 disk.rbd0.in‚0# disk.dm-0.out‚23917.05615234375 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 334.48046875$ memory_percent‚93.83840560913086 disk.rbd2.out‚ 63.142578125 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875 disk.sda1.in‚0 cpu‚25.22098253062435 disk.rbd0.out‚0& disk.nvme0n1.out‚23917.05615234375 disk.dm-0.in‚ 334.48046875 disk.sda1.out‚0 disk.dm-1.out‚ 1216.48046875 disk.drbd1777.out‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 1216.48046875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2747.66015625 disk.nvme0n1.in‚ 334.48046875 disk.drbd1777.in‚0( disk.nvme0c0n1.out‚23917.05615234375' proc.memory.availableMB‚ 1009.515625% disk./.usageGB‚1081.8985557556152 disk.sda.in‚ 0.07421875f_Iwpþbk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPÙNÀtÿbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPFeÄn€bi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JP‰W¨lbg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JP!nx ~‚by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPè!lƒbg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JP4«o„bj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JPš0Q¾t…bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPïåìy†bt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP[ªÁ×r‡bm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JP?Ogåwˆbr ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPÝŒq:y‰bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPRÃjiŠbd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JP÷ ¢ÿr‹bm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP¦«@nŒbi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JP³óvbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPïƒB‹rŽbm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JPuEMxbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPŽrÇibd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JP$êüz‘bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPZå au’bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP‘™nu“bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP*ˆ×so”bj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JPÆ©ßÆl•bg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JPë?ßq–bl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPxƒC®n—bi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JPSÁ x˜bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPDx3Lj™be BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JP}˜÷Æqšbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JPq×lÜn›bi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP¸µÇDkœbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPÉ« obj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JPdù;Yožbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPck{cŸb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JPÔVBÍt bo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPf3¡Œp¡bk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JPÒ»Ýo¢bj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JP8ÊÈgl£bg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JP6Îáq¤bl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPPZÚ“u¥bp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPº‰¸r¦bm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPˆ÷„9]§bX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JP¸¢Â.q¨bl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JP”{ï%o©bj GDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current"trainer/global_step2:JP\ ,vªbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPæ…µþk«bf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JPᬈän¬bi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP²±7y­bt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP€V-Sx®bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPé„bi¯bd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JPÄ»úÖt°bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPŠu²m±bh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JPÊ,Rgv²bq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP°õRt³bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPúŠ§“t´bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP3;@0tµbo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP)…os¶bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPZs1qe·b` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JPÊÁgôm¸bh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPEÿä5q¹bl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPLT¢­mºbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JP6T¾s»bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPå±}ér¼bm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP¼€Î9t½bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPáòsæl¾bg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JPúE)Ùt¿bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPWèwÀbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP2ÈøèsÁbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPºvZ×wÂbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPmVQymÃbh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPàù¸RvÄbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPiF»“mÅbh EDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current"trainer/global_step2:JPBáðâkÆbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JP®Ô!xÇbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPâÉHlyÈbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPfó7yÉbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPXJgeÊb` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JPî×ÕgpËbk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÛlEeÌb` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JPѸ%`dÍb_ DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JPº…*kôbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JP+.ƒ¢tõbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP©RÊsöbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPîÑSf÷ba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPphI:uøbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPˆ’Ö2ù- _wandbruntime‚1308  _runtime‚1308™rM,*ú¦T VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7033847808 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 528482304 % _timestamp‚1.7883962450449338e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30310 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2198728274770 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚54793334 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚8658039 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6651346893 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚46274264705536 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175345152 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 521397248 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚32053402 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚56109825925120 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚559 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7033847808 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1354 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2205205968896 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚13351174 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚32033603 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175345152 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6505365504 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193348096 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚48479452671488 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8658456 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2205381314048 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚559 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚22759731 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚54793334 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 528482304 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚46274246702592 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32033603 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6189583360 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚22759731 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2233158613504 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚48479646019584 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚46274264705536 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 trainer/global_step‚1208 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚56109991190016 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚45719283578706 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚45719094730694 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚19799 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚13351176 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚54813135 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚53876808636416 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚43520538264576 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2198556466118 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚307 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚307 _runtime‚1308.605209309 _step‚96 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22009632 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚559 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚252 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚48479452671488 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19801 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30310 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚419 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚32053402 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3516 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚252 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6408446976 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 165264896 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 319570432 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0÷u®·ûªT ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175345152 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6505365504 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193348096 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚48479452671488 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8658456 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2205381314048 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚559 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚22759731 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚54793334 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 528482304 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚46274246702592 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32033603 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6189583360 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚22759731 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2233158613504 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚48479646019584 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚46274264705536 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 trainer/global_step‚1208 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚56109991190016 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]JDeviceStatsMoÅ#íÁ-nitor.on_validation_batch_start/requested_bytes.all.allocated‚45719283578706 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚45719094730694 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚19799 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚13351176 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚54813135 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚53876808636416 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚43520538264576 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2198556466118 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚307 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚307 _runtime‚1308.605209309 _step‚96 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22009632 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚559 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚252 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚48479452671488 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19801 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30310 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚419 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚32053402 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3516 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚252 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6408446976 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 165264896 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 319570432 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7033847808 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 528482304 % _timestamp‚1.7883962450449338e+09 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30310 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2198728274770 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚54793334 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚8658039 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6651346893 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚46274264705536 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175345152 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 521397248 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚32053402 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚56109825925120 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚559 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7033847808 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1354 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2205205968896 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚13351174 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚32033603 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6408446976`DÞôÆzübu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPóšý´wýbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP¶Ž ánþbi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPtÖatÿbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPmFÏr€bm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP¾–ìubp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPüªÀps‚bn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP ïÞ*nƒbi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPjéöi„bd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JPÜn’„o…bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPÍ¢"‚w†br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP;g¬=|‡bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP$L)qˆbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPæSŒo‰bj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JPwJR"pŠbk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPóø„šs‹bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPü@{½rŒbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPÎçàobj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPÄVÿRtŽbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP'Átrbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP6oÙ‚rbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPPE&bo‘bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP,Ø(™u’bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPw›ñHq“bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPê¨ò/f”ba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPgC¹ìe•b` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JP"âkh–bc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JPÿAnÛj—be BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JPá¬Dc˜b^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JPÃ#¹Mm™bh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP,Χpšbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPñ­b´x›bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP"€´}jœbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JP¶æ°lbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JP 0šþwžbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPˆ–â•vŸbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPR¶âéw br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPS2òƒc¡b^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JP©¦NQm¢bh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JP™ý$ j£be BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JP-Síu¤bp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP¾~ßx¥bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP EÒKk¦bf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JPe$€úv§bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP¬Âp¨bk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPãÉÿm©bh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JP\Ð/ötªbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP1CÈbg«bb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPïfˆt¬bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP)Í™a­b\ 9DeviceStatsMonitor.on_validation_batch_end/max_split_size"trainer/global_step2:JPš4àøk®bf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPþ÷´Fm¯bh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JP( Mñi°bd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JP ºËúv±bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPС1Ùm²bh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JPµªqmq³bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JP쉽m´bh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPÖ…mâkµbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JP¾‰Rm¶bh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JPfßÎéo·bj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JPè;·g¸bb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP7@%DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPEžä lÆbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JP!°FrÇbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPdb|gÈbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JP…7‹jÉbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JP±¨ÜwÊbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP…ídËb_ DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚559 _runtime‚1309.809944416 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19801 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6873438720 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1382 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚559 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚48506779533312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚53904871590912 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚307 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚43545931107328 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚13360352 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚54817010 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30310 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2233362050048 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22024443 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚8664510 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚45745002517138 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7033847808 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6505365504 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚54817010 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚417 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193348096 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3516 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚419 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6651346893 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚307 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚252 trainer/global_step‚1208 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚54836811 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚46301035454976 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 319570432 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6189583360 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2199071409810 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32061758 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19799 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2205550730240 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 521397248 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171808652 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚48506586185216 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30535 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚22775051 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2205726075392 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30535 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚22775051 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2205726075392 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚559 _runtime‚1309.809944416 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19801 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6873438720 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1382 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚559 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚48506779533312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚53904871590912 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚307 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚43545931107328 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚13360352 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336a€ÑIRú‚(:… à…ãÔ›òæÒ gpu.0.gpu‚12.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚85.532' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.1064 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Õ)Gû‚(:¼ à…ãÔÛŠãÕ network.recv‚511024 disk.rbd2.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 1252.72265625 disk.sda.in‚ 0.07421875% disk./.usageGB‚1081.8985214233398 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 1252.72265625( disk.nvme0c0n1.out‚24015.43896484375& disk.nvme0n1.out‚24015.43896484375' proc.memory.availableMB‚ 925.8828125* proc.memory.percent‚17.279911041259766 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875 disk.sda.out‚ 1252.72265625) disk./.usagePercent‚72.60711754604368 disk.sda2.out‚0 disk.nvme0n1.in‚ 334.48046875 disk.dm-0.in‚ 334.48046875# disk.dm-0.out‚24015.43896484375 cpu‚25.265520872415802 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 334.48046875# memory_percent‚94.3488597869873 disk.drbd1777.in‚0 disk.rbd2.out‚ 63.216796875 disk.rbd1.out‚ 0.67626953125 disk.rbd0.out‚0 network.sent‚ 1.686829e+06 disk.sda1.out‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2831.140625 proc.cpu.threads‚17Û.zh;üb6 loss/val_total_loss"trainer/global_step2:JPpõ¿JýbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÅõñ NþbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÓ~ÿ¶NÿbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPpôjxN€bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP;üÒJbE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPLJñH‚bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPØ0gžJƒbE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP>½åN„bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÙlUDH…bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP0×.ÂN†bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP{T 2J‡bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¡>JˆbE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÊèH‰bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPTÂϬHŠbC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPõÜÖJ‹bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPóˆÝ­HŒbC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPè]ѳHbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP6îÒÜHŽbC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPªÃ6NbI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÃïpŒHbC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÏäJ‘bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPˆ0DJ’bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP¼¾8óJ“bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP@9¥J”bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPª"CJ•bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPcxJ–bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP lUxN—bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPl’c¦J˜bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¤­Ñ˜J™bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPN²NšbI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPŽ_EBH›bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP|«¦žJœbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÆ'ÇNbI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPD‰nÅNžbI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPûµãrNŸbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP/WIJ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPeY2N¡bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP’¢N¢bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPŽqž.H£bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPñU(bN¤bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP\¼H¥bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPoÆñN¦bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP²êÜJ§bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÿ¹µ:@¨b; particle/accuracy_1_of_6"trainer/global_step2:JPv+NÉbI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP”°DNÊbI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÞωêJËbE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP<¤»CJÌbE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP-~ {JÍbE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPà6ìjHÎbC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPÁ¹¡›8Ïb3 particle/f_score"trainer/global_step2:JPóPÿJÐbE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP7YÃñJÑbE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÇ%HHÒbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPåLNÓbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP{®²;Ôb6 jet/accuracy_4_of_4"trainer/global_step2:JPᨲÍ9Õb4 particle/accuracy"trainer/global_step2:JP®ƒÿJÖbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP6I)J×bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPn{°mNØbI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPfûñfJÙbE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP .JÚbE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPcnµNÛbI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP7wþ"HÜbC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP”F©HÝbC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP®¶RÅ@Þb; particle/accuracy_2_of_7"trainer/global_step2:JPeÿ%‹HßbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP´:°5NàbI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP«¹NábI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP"(TÈJâbE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP`îíNãbI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP”HäbC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPïe0jJåbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPØãÕÖNæbI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP.­š™;çb6 jet/accuracy_4_of_7"trainer/global_step2:JPdøœÃ;èb6 jet/accuracy_5_of_7"trainer/global_step2:JP¨ÑÒHNébI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPþkòPHêbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPHœ¸JJëbE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÿëí¶NìbI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP°À?¿NíbI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP®Ù‚¦NîbI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPþΔòNïbI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP dJðbE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPPDù¢NñbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPð1áJòbE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPR2¸HóbC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP½ãÄJôbE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŽ·éNõbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPhDô‚HöbC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPêו;÷b6 jet/accuracy_3_of_5"trainer/global_step2:JP“ZbåJøbE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP ˆdHùbC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPV—ÿJúbE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŠ±FÖJûbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP 6¥LJübE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPA¬l­HýbC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPu}#ØHþbC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPü”JÿbE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP^¶áJ€ bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPýB¬±N bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP륈ðN‚ bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP3eô<ƒ b7 particle/sensitivity"trainer/global_step2:JPpp =;„ b6 jet/accuracy_1_of_6"trainer/global_step2:JP§õcÜJ… bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP ÌÉwH† bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP& »þH‡ bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP‘âiqNˆ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPh×ÎJ‰ bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP4IÌONŠ bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPu-fTH‹ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¥wvJŒ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÕ”žN bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPê<,æNŽ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPKNh:J bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‘y bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¯ UJ‘ bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÑczH’ bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPñÐaJ“ bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPéK6ÑH” bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP$­é³N• bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP–Ó%,H– bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPé.ù“H— bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPÌï¥}J˜ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPmÀH™ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPꜺêJš bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP’ê÷N› bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP«V»XNœ bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPˆ."RN bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÀ]X¨Jž bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPºÛDNŸ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP7Á=ëN  bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÍ}@¡ b; particle/accuracy_1_of_4"trainer/global_step2:JPc*8xJ¢ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPY$±þJ£ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPØOІN¤ bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÖ|67@¥ b; particle/accuracy_2_of_4"trainer/global_step2:JP|1™J¦ bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPµ4£@§ b; particle/accuracy_4_of_6"trainer/global_step2:JPÆëÑ}J¨ bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP¡[0 J© bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP,Õ¼íNª bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPd5CSN« bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP9³šªH¬ bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÞœ8n@­ b; particle/accuracy_3_of_3"trainer/global_step2:JPÇnرJ® bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÓÏH¯ bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPsùöJ° bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP¡ô[J± bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPˆ€@N² bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP‡NßN³ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPy1J´ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÄÓSHµ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP#ÊU¹J¶ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP‚9µ9J· bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPf ç™N¸ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP•ŸJ¹ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP«5uiJº bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP(ŽrJ» bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP]‡¤àJ¼ bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP"Ûr0J½ bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP¹Õ†ÀJ¾ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPë¤$Ë@¿ b; particle/accuracy_1_of_7"trainer/global_step2:JPÄó½Ô@À b; particle/accuracy_4_of_7"trainer/global_step2:JP±’‡Ê@Á b; particle/accuracy_5_of_7"trainer/global_step2:JPzüO_J bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPDå¾q;à b6 jet/accuracy_2_of_3"trainer/global_step2:JPš»HÄ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP¶£HÅ bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÆ÷0JÆ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÁÕ´yHÇ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP½\3Q@È b; particle/accuracy_4_of_4"trainer/global_step2:JPÖw«ëJÉ bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP¬¸üJÊ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPö¶ÆÕHË bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÌÝ4HÌ bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¬?ÕkJÍ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP<À}™@Î b; particle/accuracy_6_of_7"trainer/global_step2:JPÈéJÏ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP˜j“NÐ bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP\åêöNÑ bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPS…NÒ bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPdÅJÓ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÂŹ¼JÔ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPcêÕHÕ bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP@F+öNÖ bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPäƒò´N× bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÚ ëNØ bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPJ BHÙ bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP°×ÜÅHÚ bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP}”ÃyHÛ bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPøÉ°îNÜ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÏÿïJÝ bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP'°ÄHÞ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÛD!¼Jß bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¹ÃoöNà bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPíùVA@á b; particle/accuracy_7_of_7"trainer/global_step2:JPO <óJâ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP";ê£Nã bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPîƒVnNä bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPDdÆNå bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPµ?!UHæ bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP,[Hç bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPVÅ‘Nè bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPσ>àNé bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP~ÿífNê bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP=ìJë bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPmv*Hì bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP$×fJí bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¾Á™\Hî bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPö¨i)Nï bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPR%ƒNð bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPlÚNñ bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP™ú˜ÏNò bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPU)>!Hó bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP ó†;ô b6 jet/accuracy_1_of_7"trainer/global_step2:JPåQŒHõ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPdð¬CHö bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPôrè@÷ b; particle/accuracy_4_of_5"trainer/global_step2:JP>†…zHø bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP júHù bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP²”Jú bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPXÄïJû bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP“´7[Nü bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPž)…ŸNý bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP]E¤hJþ bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPFUäëJÿ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPµ°NJ€ bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPpïáœN bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP0C$J‚ bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPòȸc@ƒ b; particle/accuracy_5_of_6"trainer/global_step2:JP:3ÒëJ„ bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPšh~$H… bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP8–í‘J† bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP“%ˆPH‡ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP§šÙ`Hˆ bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPaÚC¶N‰ bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP´@GSNŠ bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPšœhN‹ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP&g» HŒ bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÞ¡”ûJ bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP9ݲHŽ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPR»×ZH bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP¹ÝÈH bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPB†ÓN‘ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP“GØJ’ bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPõ;,J“ bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÏÔMñJ” bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÀ ­N• bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP§Ð*AH– bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÅ–´‡J— bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPc6ÕH˜ bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPÎJ|H™ bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP:—¾Hš bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPŸZÅÇN› bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPpZãHœ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPH¶ž; b6 jet/accuracy_5_of_5"trainer/global_step2:JP19iJž bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP RšùJŸ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP03YJJ  bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÀäîµJ¡ bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP%ÂêJ¢ bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPfÿ,£ bC Purity/1FRt*SRqqt3FBt/F™K‰*Rt_purity"trainer/global_step2:JPlÄ}J¤ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP²´ 0H¥ bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPI1“N¦ bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPë‰îaH§ bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP+ÀWžN¨ bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPþ‹;H© bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPkªåaHª bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPw©¯N« bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP%cÀ N¬ bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPîd0J­ bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPCôü®J® bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¹QN¯ bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPk¯{J° bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPœ>ä2H± bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP_îR@² b; particle/accuracy_3_of_6"trainer/global_step2:JP¶IGäJ³ bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPFkÔ#H´ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP RÉNµ bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÇYº´J¶ bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP„Ö:J· bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP ¤îJ¸ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPž3’aJ¹ bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP$|j;º b6 jet/accuracy_3_of_7"trainer/global_step2:JPI'˜¬J» bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPú¡éPJ¼ bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP=JN½ bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP£ˆ±;¾ b6 jet/accuracy_1_of_2"trainer/global_step2:JPmÔN»H¿ bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP¢ s(JÀ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP*p,wNÁ bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP¡}½J bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPðð|@à b; particle/accuracy_2_of_6"trainer/global_step2:JPÿ/ÄCJÄ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPØ2ç]NÅ bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPê}áNÆ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPQ®iàNÇ bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP®…Ê^JÈ bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPL†vÄNÉ bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPá¯dMJÊ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¤^ìbHË bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP³OJÌ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPC uJÍ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¸ï”lNÎ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPíg  NÏ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP+Iá6JÐ bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPà[JÑ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP91þ%JÒ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPmuµ„JÓ bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPv_HÔ bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP\U&iJÕ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPe¬8JÖ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPTd°bJ× bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPœ@9ëJØ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÛÕiHÙ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP}1³?HÚ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPe•Ò\NÛ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPPSNÜ bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPšÈRª;Ý b6 jet/accuracy_2_of_5"trainer/global_step2:JP?UÆHÞ bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPà’î Nß bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPa”gZNà bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPZáNá bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP!©Ø;â b6 jet/accuracy_2_of_2"trainer/global_step2:JPø‹DHã bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPä«fdHä bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPüm~ÙJå bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPŸÄ©Næ bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPãËþíJç bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPA.âJè bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP[”,Jé bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP)˜¯,Hê bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP–{+sNë bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP„LHì bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÊ@ƒwNí bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP&/QüHî bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP5f&ŸJï bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP˜³æaNð bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPXç¢Nñ bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPdˆ¤Jò bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÍÇöHó bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP¡µÛ¤Hô bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPî½Ú¤Hõ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÞ-ÈYHö bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPž~ËþN÷ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPBSnõJø bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÏѯNù bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP¯s€$Hú bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP{Ö;û b6 jet/accuracy_1_of_4"trainer/global_step2:JP–Œ…Jü bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPðvØMHý bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¦å©„Jþ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP&ÿøHÿ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÐ\1N€ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP…žh}N bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPa—Æ°J‚ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¬é5ÔHƒ bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP.ìÜJ„ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPð¬éÅN… bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÑ#K¸J† bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP) MlJ‡ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¹ÝÅÆNˆ bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÁ­Þ“H‰ bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPoÖùHŠ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPêØþîH‹ bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPÞPHŒ bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÆÇJ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP3Èb½NŽ bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP¹N‘«J bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPnû&N bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP<:)mJ‘ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPAJ­H’ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPLt|ÏN“ bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPçH)N” bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÞÝ7fN• bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPРH– bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP;XÝJJ— bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP‡–tÂJ˜ bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP KQJ™ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÝb« Jš bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP\YߣJ› bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPDûÕ|Jœ bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPî1W0J bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPÝo%ÏJž bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¡F†lNŸ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¨ÈûœN  bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP5O¤J¡ bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP!‘PN¢ bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP3³O@£ b; particle/accuracy_1_of_5"trainer/global_step2:JP;ŽãJ¤ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP‹½$|J¥ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPëIN¦ bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPzñ®šN§ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPø<€@J¨ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÉWœJ© bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPƒ*q Hª bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPe„ŽªH« bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP:å‚©N¬ bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP´,î-J­ bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÝ~H¸N® bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÀ&)H¯ bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP4È— @° b; particle/accuracy_3_of_4"trainer/global_step2:JP²Q¢1@± b; particle/accuracy_3_of_7"trainer/global_step2:JP×]J² bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPn ,H³ bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP€[æ–J´ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP9hë7Hµ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPa •®J¶ bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPòÜœkJ· bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÏ\|ìN¸ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÚóZ³N¹ bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPªJ»°Hº bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPPïЖJ» bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPIp@+J¼ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPï°¬J½ bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPloªEN¾ bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPC~†¹H¿ bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPCYóHÀ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP3ª‡bJÁ bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP‰ ŠcJ bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPýîâHà bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPd˜³vNÄ bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPâ©ÝöNÅ bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPú jNÆ bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPâ¤gÏNÇ bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP飛âNÈ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPN ¸8HÉ bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP‘etÜJÊ bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPwC¦QJË bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPˆ4/NÌ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPSÄÒNÍ bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÃÈÖ/HÎ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPQ­ç™JÏ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP¸œzJÐ bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP}øbKNÑ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP¼åƒŒNÒ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP¢6²†HÓ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPÁ7¶MJÔ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP=ÈJÕ bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPdž …HÖ bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP£þ&=@× b; particle/accuracy_1_of_3"trainer/global_step2:JPQrÿ!HØ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPØFÚHÙ bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPÌó;dHÚ bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP}TZ£HÛ bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPH_/›HÜ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP´ÆWBNÝ bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPèâÅHÞ bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP=þ4JNß bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPD›¹Hà bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPõ¥o5Ná bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP»!žFJâ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPeI)Gã bB validation_average_jet_accuracy"trainer/global_step2:JPÁ™Jä bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPB bÁHå bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÎÞg»Jæ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÆž€i@ç b; particle/accuracy_1_of_1"trainer/global_step2:JP2뙑Jè bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPR„­ Jé bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP.}ŸNê bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP.L ÁNë bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP¡¿Š&Jì bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP´‚U¨Hí bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÁšNî bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPj`_‰Hï bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP™ðØ_Jð bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP—L]ñHñ bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPAÒþ¶Hò bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPÍÇëøNó bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPsk¬Jô bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP|¨g®Jõ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPJI"IHö bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPæƒÒÁJ÷ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP­ïbNø bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPŒcÕ”Hù bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPzô§yHú bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPü܉öNû bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPE ÚI;ü b6 jet/accuracy_5_of_6"trainer/global_step2:JP忘Hý bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP7nHþ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPš¬eHÿ bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP|RÚoH€ bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPwÛ¬þJ bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŽýÚH‚ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP7ϤNƒ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPjþ*ñ;„ b6 jet/accuracy_1_of_3"trainer/global_step2:JPÂBExH… bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP/ÁÃQJ† bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPCH‡ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP»|ÏFJˆ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP~2òN‰ bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP«¨™NŠ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP┆N‹ bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP€̲NŒ bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP«ÊžcJ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPI…mæHŽ bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP–;GrH bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP^0#@N bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPA)†J‘ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPB³oH’ bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPGR J“ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP~Yø!J” bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPüéN• bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPlGÊÐN– bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPøB<"N— bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP‘utjJ˜ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP<@\^J™ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPèÑ|Jš bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPa‚ÑìH› bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP+ÅÌJœ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP²²«J bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPre>BNž bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP8„d¸HŸ bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPgùß J  bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP÷IÏ0N¡ bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP´Ñ²fJ¢ bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP~Ý6&N£ bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPc·¶H¤ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPÆfH¥ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPÅl—H¦ bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÑ(ßH§ bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPFW8J¨ bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP"è‚^N© bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP¿Öy\Hª bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPt•Ô›J« bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPHXÍNÜ bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÆr@Ý b; particle/accuracy_2_of_3"trainer/global_step2:JP?âË HÞ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP›ý0ñJß bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP8“ÏlHà bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPñôNá bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPz2œmNâ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÝ‘Hã bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP½n9Jä bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPÑäE>Nå bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP4¶ÀNæ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPö©?gJç bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP5-NHè bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP8iKHé bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPRîì·Hê bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPxÑ”+Hë bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPdNì bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPɸÆ&Ní bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPGÞfrNî bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPf¹õïNï bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPì¢ÎUNð bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP®ÙlÀJñ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP¼¼XHò bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÔÁQNó bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP½{aNô bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPž@ôøNõ bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP÷¥¤;ö b6 jet/accuracy_4_of_5"trainer/global_step2:JPlç'JÇ bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP¦«„;È b6 jet/accuracy_6_of_7"trainer/global_step2:JP”wSJÉ bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPøj‹SNÊ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPä˜Î2Ë - _wandbruntime‚1325  _runtime‚1325 BüzÌ Š£ ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.15625 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9952381253242493 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964123964309692 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9661323428153992 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.025252947583794594 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2532571256160736 7validation_average_jet_accuracy‚0.25673872232437134 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.06949318200349808 particle/accuracy_1_of_1‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.08355230838060379 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.11764705926179886 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.06759259849786758 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.019246364012360573 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9849250912666321 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.21620701253414154 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.16668617725372314 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.5838574767112732 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.008620689623057842 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.0479760468006134 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03413573279976845 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.04320988059043884 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.07947531342506409 trainer/global_step‚1208 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.05844241380691528 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.10631164163351059 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.08143109828233719 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022045858204364777 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.09878747910261154 +jet/accuracy_2_of_6‚0.20512819290161133 jet/accuracy_4_of_6‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.03695547953248024 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.0809386819601059 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9806202054023743 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.07687687128782272 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.027103925123810768 +particle/specificity‚0.9121119379997253 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.09953703731298447 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.27771803736686707 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21682126820087433 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.04885057732462883 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.14429756999015808 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.1875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.015189237892627716 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9901649355888367 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.193452388048172 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_3‚0.00451900577172637 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.5996207594871521 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.1036888062953949 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.07109788805246353 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9869351387023926 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990662932395935 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9600694179534912 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.08507765084505081 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1587201952934265 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_5‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.1111111119389534 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.1593831181526184 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870532751083374 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.02916037291288376 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2110215127468109 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.10468748956918716 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.15000000596046448 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.08482609689235687 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.15229882299900055 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.02880658395588398 particle/accuracy_2_of_2‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ="Purity/*FRt2SRqqt*FBt/event_purity‚0.00042900044354610145 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.05808081105351448 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.06034482643008232 _step‚98 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9821085333824158 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03260841965675354 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.01923076994717121 9 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.048572104424238205 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.08281927555799484 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.08796297013759613 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.017307482659816742 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.09848006069660187 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0004960317746736109 9 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.038470786064863205 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.014639372006058693 particle/accuracy_6_of_6‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.048634450882673264 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 )loss/val_total_loss‚70.54165607269982 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.33271339535713196 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.2719298303127289 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3332569897174835 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9791666865348816 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1458333283662796 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.25 & Purity/2FRt4SRqqt0FBt/FRt_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9903758764266968 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9868083000183105 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.0833333358168602 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.05389384925365448 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.11449404060840607 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.04082821309566498 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9894823431968689 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 jet/accuracy_7_of_7‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.06531532108783722 jet/accuracy_2_of_7‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.037911586463451385 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.6983230113983154 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.0416666679084301 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.1428571492433548 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.028575841337442398 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt0SRqqt1FBt/event_purity‚0.021379617974162102 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9968253970146179 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196217060089111 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0524197481572628 9 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.051645293831825256 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_6_of_6‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FRt_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.10557089000940323 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862207770347595 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.1196078509092331 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 &particle/f_score‚0.220880389213562 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.029565224424004555 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_4‚0 'particle/accuracy‚0.798301100730896 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.004032257944345474 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.16472980380058289 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.142361119389534 & Purity/2FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9885208010673523 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.21011903882026672 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 jet/accuracy_4_of_7‚0 jet/accuracy_5_of_7‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0997023731470108 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.810110330581665 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.060942765325307846 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.125 ,jet/accuracy_3_of_5‚0.016577061265707016 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.02500000037252903 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 _runtime‚1325.479516031 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2047881782054901 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0004810004902537912 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.15323448181152344 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.18318668007850647 *jet/accuracy_1_of_6‚0.7756410837173462 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9805417060852051 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993759274482727 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.11337561160326004 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.25757578015327454 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.16967636346817017 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.10659721493721008 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583134651184 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.5996404886245728 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2413194328546524 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2015114277601242 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 particle/accuracy_4_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.07407407462596893 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9747536182403564 particle/accuracy_3_of_3‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.03999999910593033 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.00788161251693964 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.14546145498752594 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.0763888880610466 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.08965732157230377 particle/accuracy_1_of_7‚1 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.035234156996011734 *jet/accuracy_2_of_3‚0.0893213301897049 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9892781972885132 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.09852810204029083 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.03256705030798912 particle/accuracy_4_of_4‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9941798448562622 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.08328632265329361 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.12082358449697495 particle/accuracy_6_of_7‚1 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.010310264304280281 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt0SRqqt0FBt/FRt_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.07894736528396606 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.08839793503284454 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9819600582122803 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.24468861520290375 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.09375 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.985714316368103 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0 % _timestamp‚1.7883962619173498e+09 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.1940789520740509 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.03277330845594406 particle/accuracy_4_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030370397493243217 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.20867635309696198 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.12962962687015533 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ;"Purity/1FRt0SRqqt2FBt/event_purity‚0.022040074691176414 particle/accuracy_5_of_6‚1 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.0833333358168602 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955291748046875 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07333333790302277 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/0FRt1SRqqt1FBt/event_purity‚0.055655092000961304 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.0852833241224289 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026234569028019905 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.12734206020832062 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.050624504685401917 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.08873670548200607 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.05028623715043068 jet/accuracy_5_of_5‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1552591621875763 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.1887976974248886 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.0833333358168602 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.09964901208877563 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.15794573724269867 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.0555555559694767 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.14203408360481262 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.13682912290096283 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.11337561160326004 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19396549463272095 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_3_of_7‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12787014245986938 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_1_of_2‚0.4841591417789459 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.008870450779795647 particle/accuracy_2_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 epoch‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.04967005178332329 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.09773583710193634 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05383933335542679 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.09602732211351395 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.12297908961772919 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.01832648366689682 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.11661195009946823 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2142857164144516 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.12497049570083618 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.15146566927433014 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.05571698024868965 ,&PuriÚs;Åsty/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.04215604439377785 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.10106369853019714 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2887222170829773 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.964285671710968 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.25757578015327454 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988245964050293 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965113997459412 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.04711853712797165 )jet/accuracy_1_of_4‚0.636972963809967 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.10110229253768921 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.12425769865512848 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9911279678344727 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.20000000298023224 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.09986821562051773 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.10081667453050613 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9970535635948181 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.2248852699995041 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 <"Purity/0FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.01832648366689682 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.1487196385860443 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.19800816476345062 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.105064257979393 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_5‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.0694444477558136 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.015386585146188736 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.2094283550977707 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.016531242057681084 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.06060606241226196 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9932471513748169 particle/accuracy_3_of_4‚1 particle/accuracy_3_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1041666716337204 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.049467504024505615 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0003779289545491338 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.12114521116018295 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2824247181415558 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1041666716337204 9 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.052612435072660446 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21917563676834106 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0qÓâãhÍ Ž£ ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.14429756999015808 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.1875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.015189237892627716 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9901649355888367 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.193452388048172 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_3‚0.00451900577172637 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.5996207594871521 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.1036888062953949 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.07109788805246353 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9869351387023926 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990662932395935 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9600694179534912 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.08507765084505081 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1587201952934265 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_5‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.1111111119389534 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.1593831181526184 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870532751083374 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.02916037291288376 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2110215127468109 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.10468748956918716 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.15000000596046448 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.08482609689235687 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.15229882299900055 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.02880658395588398 particle/accuracy_2_of_2‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ="Purity/*FRt2SRqqt*FBt/event_purity‚0.00042900044354610145 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.05808081105351448 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.06034482643008232 _step‚98 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9821085333824158 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03260841965675354 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.01923076994717121 9 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.048572104424238205 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.08281927555799484 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.08796297013759613 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.017307482659816742 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.09848006069660187 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0004960317746736109 9 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.038470786064863205 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.014639372006058693 particle/accuracy_6_of_6‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.048634450882673264 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 )loss/val_total_loss‚70.54165607269982 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.33271339535713196 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.2719298303127289 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3332569897174835 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9791666865348816 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1458333283662796 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.25 & Purity/2FRt4SRqqt0FBt/FRt_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9903758764266968 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9868083000183105 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.0833333358168602 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.05389384925365448 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.11449404060840607 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.04082821309566498 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9894823431968689 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 jet/accuracy_7_of_7‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.06531532108783722 jet/accuracy_2_of_7‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.037911586463451385 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.6983230113983154 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.0416666679084301 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.1428571492433548 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.028575841337442398 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt0SRqqt1FBt/event_purity‚0.021379617974162102 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9968253970146179 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196217060089111 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0524197481572628 9 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.051645293831825256 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_6_of_6‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FRt_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.10557089000940323 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862207770347595 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.1196078509092331 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 &particle/f_score‚0.220880389213562 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.029565224424004555 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_4‚0 'particle/accuracy‚0.798301100730896 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.004032257944345474 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.16472980380058289 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.142361119389534 & Purity/2FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9885208010673523 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.21011903882026672 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 jet/accuracy_4_of_7‚0 jet/accuracy_5_of_7‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0997023731470108 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.810110330581665 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.060942765325307846 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.125 ,jet/accuracy_3_of_5‚0.016577061265707016 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.02500000037252903 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 _runtime‚1325.479516031 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2047881782054901 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0004810004902537912 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.15323448181152344 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.18318668007850647 *jet/accuracy_1_of_6‚0.7756410837173462 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9805417060852051 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.993759274482727 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.11337561160326004 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.25757578015327454 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.16967636346817017 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.10659721493721008 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583134651184 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.5996404886245728 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2413194328546524 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2015114277601242 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 particle/accuracy_4_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.07407407462596893 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9747536182403564 particle/accuracy_3_of_3‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.03999999910593033 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.00788161251693964 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.14546145498752594 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.0763888880610466 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.08965732157230377 particle/accuracy_1_of_7‚1 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.035234156996011734 *jet/accuracy_2_of_3‚0.0893213301897049 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9892781972885132 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.09852810204029083 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.03256705030798912 particle/accuracy_4_of_4‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9941798448562622 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.08328632265329361 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.12082358449697495 particle/accuracy_6_of_7‚1 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.010310264304280281 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt0SRqqt0FBt/FRt_purity‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.07894736528396606 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.08839793503284454 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9819600582122803 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.24468861520290375 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 ("Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.09375 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.985714316368103 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0 % _timestamp‚1.7883962619173498e+09 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.1940789520740509 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.03277330845594406 particle/accuracy_4_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030370397493243217 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.20867635309696198 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.12962962687015533 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ;"Purity/1FRt0SRqqt2FBt/event_purity‚0.022040074691176414 particle/accuracy_5_of_6‚1 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.0833333358168602 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955291748046875 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07333333790302277 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/0FRt1SRqqt1FBt/event_purity‚0.055655092000961304 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.0852833241224289 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.026234569028019905 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.12734206020832062 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.050624504685401917 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.08873670548200607 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.05028623715043068 jet/accuracy_5_of_5‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1552591621875763 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.1887976974248886 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.0833333358168602 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.09964901208877563 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.15794573724269867 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.0555555559694767 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.14203408360481262 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.13682912290096283 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.11337561160326004 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19396549463272095 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_3_of_7‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12787014245986938 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_1_of_2‚0.4841591417789459 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.008870450779795647 particle/accuracy_2_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 epoch‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.04967005178332329 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.09773583710193634 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.05383933335542679 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.09602732211351395 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.12297908961772919 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.01832648366689682 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.11661195009946823 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2142857164144516 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.12497049570083618 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.15146566927433014 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.05571698024868965 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.04215604439377785 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.10106369853019714 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2887222170829773 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.964285671710968 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt*SRqqt2FBt/SRqqt_purK Ó)ity‚0.25757578015327454 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988245964050293 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965113997459412 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.04711853712797165 )jet/accuracy_1_of_4‚0.636972963809967 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.10110229253768921 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.12425769865512848 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9911279678344727 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.20000000298023224 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.09986821562051773 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.10081667453050613 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9970535635948181 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.2248852699995041 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 <"Purity/0FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.01832648366689682 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.1487196385860443 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.19800816476345062 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.105064257979393 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_5‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.0694444477558136 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.015386585146188736 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.2094283550977707 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.016531242057681084 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.06060606241226196 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9932471513748169 particle/accuracy_3_of_4‚1 particle/accuracy_3_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1041666716337204 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.049467504024505615 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0003779289545491338 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.12114521116018295 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2824247181415558 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1041666716337204 9 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.052612435072660446 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21917563676834106 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.15625 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9952381253242493 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964123964309692 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9661323428153992 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.025252947583794594 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2532571256160736 7validation_average_jet_accuracy‚0.25673872232437134 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.06949318200349808 particle/accuracy_1_of_1‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.08355230838060379 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.11764705926179886 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.06759259849786758 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.019246364012360573 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9849250912666321 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.21620701253414154 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.16668617725372314 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.5838574767112732 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.008620689623057842 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.0479760468006134 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03413573279976845 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.0416666679084301 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.04320988059043884 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.07947531342506409 trainer/global_step‚1208 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.05844241380691528 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.10631164163351059 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.08143109828233719 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.022045858204364777 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.09878747910261154 +jet/accuracy_2_of_6‚0.20512819290161133 jet/accuracy_4_of_6‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.03695547953248024 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.0809386819601059 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9806202054023743 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.07687687128782272 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.027103925123810768 +particle/specificity‚0.9121119379997253 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.09953703731298447 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.27771803736686707 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21682126820087433 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.04885057732462883 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146bX“$á›Î ‚(: ï…ãÔ“Œ›Ó gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.531' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.906200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÐÖ'Ž?Ï ‚(:´ ï…ãÔ„¸ŸÖ disk.nvme0n1.in‚ 334.48046875 disk.dm-0.in‚ 334.48046875 disk.drbd1777.in‚0( proc.memory.availableMB‚ 563.50390625 network.sent‚ 1.75675e+06 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875# disk.nvme0n1.out‚24122.41796875 disk.drbd1777.out‚0* proc.memory.percent‚17.397069931030273 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 1287.47265625 disk.sda.out‚ 1287.47265625$ memory_percent‚96.56064510345459$ disk./.usageGB‚1082.231674194336 disk.sda3.out‚ 1287.47265625% disk.nvme0c0n1.out‚24122.41796875 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0) disk./.usagePercent‚72.62947570803901 disk.rbd2.in‚0 disk.rbd2.out‚ 64.853515625 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.dm-0.out‚24122.41796875 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 cpu‚25.280754887276863 disk.rbd1.out‚ 0.67626953125" proc.memory.rssMB‚ 2850.3359375 disk.rbd3.out‚0 network.recv‚516355" disk.nvme0c0n1.in‚ 334.48046875K=)›Ð ‚(: þ…ãÔÏ—“Ó gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.177000000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Ü_Q/Ñ ‚(:¤ þ…ãÔˆŠÃÖ disk.nvme0n1.in‚ 334.48046875 disk.rbd3.out‚0) disk./.usagePercent‚72.62947442800134 disk.drbd1777.out‚0 disk.sda3.out‚ 1323.859375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 334.48046875 disk.nvme0n1.out‚ 24455.8125 disk.sda2.in‚0 network.recv‚522626 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 0.07421875 disk.sda.out‚ 1323.859375# proc.memory.rssMB‚ 2850.33984375 network.sent‚ 1.880088e+06 disk.rbd1.out‚ 0.67626953125 disk.dm-1.in‚ 0.07421875$ memory_percent‚96.56414985656738* proc.memory.percent‚17.397093772888184 disk.rbd2.out‚ 64.869140625 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875! disk.nvme0c0n1.out‚ 24455.8125 disk.dm-1.out‚ 1323.859375 disk.rbd0.in‚0 disk.dm-0.out‚ 24455.8125% disk./.usageGB‚1082.2316551208496 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 334.48046875' proc.memory.availableMB‚ 562.9296875 cpu‚25.278229522855217®9%›Ò ‚(: †ãÔœÛðÒ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.828201293945312) gpu.0.memoryAllocatedBytes‚ 7583367168 gpu.0.temp‚28.0 gpu.0.powerWatts‚75.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.133600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08&ÇêBÓ ‚(:· †ãÔôüØÕ disk.sda.in‚ 0.07421875 disk.sda3.out‚ 1358.71875( proc.memory.availableMB‚ 562.62109375* proc.memory.percent‚17.397093772888184( disk.nvme0c0n1.out‚24554.30615234375 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 1358.71875 disk.rbd0.in‚0% disk./.usageGB‚1082.2316551208496 disk.sda.out‚ 1358.71875" disk.nvme0c0n1.in‚ 334.48046875 network.sent‚ 1.885756e+06# disk.dm-0.out‚24554.30615234375 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.67626953125& disk.nvme0n1.out‚24554.30615234375 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚96.56603336334229) disk./.usagePercent‚72.62947442800134 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 network.recv‚526463 disk.nvme0n1.in‚ 334.48046875 disk.rbd2.out‚ 64.931640625 disk.rbd3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2850.33984375 proc.cpu.threads‚17 disk.drbd1777.out‚0 cpu‚25.275980850440945 disk.dm-0.in‚ 334.48046875ËÔÞb2Ô - _wandbruntime‚1372  _runtime‚1372-¹oÈÕ Â _step‚99 ' lr-AdamW/pg1‚0.00022300660856844893 ' lr-AdamW/pg2‚0.00022300660856844893 trainer/global_step‚1249 % _timestamp‚1.7883963087366874e+09 _runtime‚1372.296591376ùE¡ÌÖ Æ _runtime‚1372.296591376 _step‚99 ' lr-AdamW/pg1‚0.00022300660856844893 ' lr-AdamW/pg2‚0.00022300660856844893 trainer/global_step‚1249 % _timestamp‚1.7883963087366874e+09c¿£G(2× - _wandbruntime‚1372  _runtime‚1372uÕ7Æ'Ø ÀO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56004870327296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2311317842432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13905758 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33193287 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2275696191531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48134980832768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9015746 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45272884286464 % _timestamp‚1.7883963087375548e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2282357151744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47548580477995 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50417589572096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2282590736384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56920840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50417589572096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2275467221687 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58316188169728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2311052304896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56894658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23701373 trainer/global_step‚1249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22920816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58315898692096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50417337984512 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48134980832768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2282590736384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56004846387200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13905760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2282357151744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48134998835712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22921506 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 _step‚100 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23701373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48134998835712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47548334468791 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 _runtime‚1372.297591201 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33219467 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚50417337984512 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584õr£PÙ ÄO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9015058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289477632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265537536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33219467 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33193287 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50417589572096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2275467221687 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58316188169728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2311052304896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚252 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56894658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23701373 trainer/global_step‚1249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22920816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7033847808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58315898692096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50417337984512 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48134980832768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2282590736384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 528482304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56004846387200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13905760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2282357151744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48134998835712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22921506 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 _step‚100 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23701373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48134998835712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47548334468791 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 _runtime‚1372.297591201 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33219467 TADevice¢AÆz StatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚50417337984512 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56004870327296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2311317842432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13905758 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33193287 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2275696191531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 528482304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48134980832768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9015746 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚559 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45272884286464 % _timestamp‚1.7883963087375548e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2282357151744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47548580477995 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚252 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50417589572096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2282590736384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56920840 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56965120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23719552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2277270955127 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48172141125632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33245568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 % _timestamp‚1.7883963097345033e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56965120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23719552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2284166300160 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47585064895607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13916413 _runtime‚1373.295528793 _step‚101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2312896608256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22938915 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45307810979840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22938208 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58361413466624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291561472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58361121905152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50456289422848 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33219362 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23719550 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56938912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48172123122688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2313164229632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56938912 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33245568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33219362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9022502 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9021797\¢îkÔ&Ü ÎM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2284399898112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚559 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22938208 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58361413466624 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291561472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58361121905152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50456289422848 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33219362 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23719550 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56938912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48172123122688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2313164229632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56938912 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33245568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33219362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9022502 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9021797 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56965120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23719552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2277270955127 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48172141125632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33245568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚252 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 % _timestamp‚1.7883963097345033e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56965120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23719552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2284166300160 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47585064895607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13916413 _runtime‚1373.295528793 _step‚101 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 528482304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2312896608256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22938915 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45307810979840 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49992095328768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 _step‚104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2372828986368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24610321 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14438410 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2373062571008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58173791872000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49992095328768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2402902734848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2373062571008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49385115410003 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2372828986368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚753 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9359652 % _timestamp‚1.7883963621259148e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23797309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259246080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34500389 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283186176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24610321 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49992113331712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58173815812096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52364924315136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49992095328768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 _step‚104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2372828986368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24610321 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14438410 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2373062571008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58173791872000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49992095328768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2402902734848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2373062571008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49385115410003 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2372828986368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚753 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9359652 % _timestamp‚1.7883963621259148e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23797309 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259246080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34500389 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283186176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚252 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24610321 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49992113331712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚559 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7033847808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 528482304 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58173815812096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚559 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚559 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52364924315136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59154964 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60621639934976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2367471218335 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59181172 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23815812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34552670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2404732580352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34552670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 _runtime‚1426.705157554 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24628500 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9366034 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287367168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58217194721792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2374871732736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23815095 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50029237618688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59154964 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52403875753472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14449061 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34526464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2367700188387 % _timestamp‚1.7883963631441963e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24628502 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433mØ@ ð ÎM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚252 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59181172 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23815812 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 528482304 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34552670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2404732580352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34552670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 _runtime‚1426.705157554 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24628500 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9366034 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287367168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58217194721792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2374871732736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23815095 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 528482304 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50029237618688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59154964 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52403875753472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14449061 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚559 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34526464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2367700188387 % _timestamp‚1.7883963631441963e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24628502 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59181172 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7033847808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47054145648640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚252 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52404127354368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50029237618688 trainer/global_step‚1299 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2374638134784 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚559 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59154964 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60621639934976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2367471218335 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7033847808 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25519271 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2494444371456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60342737356800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14971058 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚749 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54312762233344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51849227827712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51221404332807 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35833671 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51221650342011 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24675564 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51849209824768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2463534405632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2455867748103 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35807491 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2463534405632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60342761296896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24676313 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 _step‚108 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25519273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35807491 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2463300820992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9705253 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48765536584704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9704506 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2456096717947 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271828992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54312510645760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51849227827712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295769088 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62836909899264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48765553624064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14971060 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35833671 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54312510645760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61326762 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51849209824768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61352944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25519273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806Ê—0Ðÿ ÄO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24675564 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51849209824768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2463534405632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2455867748103 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35807491 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2463534405632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60342761296896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24676313 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 _step‚108 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25519273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35807491 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2463300820992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9705253 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48765536584704 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9704506 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2456096717947 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271828992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54312510645760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51849227827712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295769088 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62836909899264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48765553624064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14971060 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35833671 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54312510645760 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61326762 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51849209824768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61352944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25519273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1478.706809726 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25519271 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2494444371456 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408Ÿlú 446976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60342737356800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14971058 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚749 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54312762233344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51849227827712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51221404332807 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35833671 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51221650342011 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253l_š&2€- _wandbruntime‚1479  _runtime‚1479t" ›Ï&ÉM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2457671481543 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62882405428736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54351713684992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2465343567360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60386116266496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2457900451595 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25537452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2465343567360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2465109969408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 _step‚109 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35859772 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51258134759623 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62882115964416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1349 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2465109969408 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54351713684992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25537450 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35833566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48800480317440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35859772 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51886352114688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24693643 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51886352114688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2495999697920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚723 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9711207 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61397224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35833566 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14981713 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25537450 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _runtime‚1479.732247961 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61371016 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51258380769035 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61397224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25537452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚246929180è˜ÇÓ&‚ÍM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60386140206592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54351462084096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62882405428736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54351713684992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2465343567360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60386116266496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2457900451595 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25537452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2465343567360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2465109969408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 _step‚109 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35859772 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51258134759623 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62882115964416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1349 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2465109969408 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54351713684992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25537450 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35833566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48800480317440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35859772 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51886352114688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24693643 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51886352114688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2495999697920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚723 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9711207 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61397224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35833566 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14981713 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25537450 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _runtime‚1479.732247961 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61371016 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51258380769035 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61397224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25537452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24692918 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2457671481543 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53706324320768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63568996 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2585891379200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2553772655616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26428223 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56260096976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15503708 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50511888292864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62511682841600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53058185274019 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10051009 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37114593 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53706342323712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 297866240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2546068011311 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56260348563968 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26428223 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25554015 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10050307 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2554006240256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15503710 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53057939264815 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53706324320768 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63568996 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2585891379200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2553772655616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26428223 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56260096976384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15503708 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50511888292864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62511682841600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53058185274019 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10051009 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37114593 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53706342323712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 297866240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2546068011311 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56260348563968 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26428223 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25554015 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10050307 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2554006240256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15503710 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53057939264815 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25571984 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65142795717632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50546797946880 _step‚113 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53743484613632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53743484613632 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63613276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26446400 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10057621 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62555085691392 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232768 trainer/global_step‚1399 % _timestamp‚1.7883964696576245e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26446402 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10056956 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2555581804032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26446400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37140668 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25571317 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 _runtime‚1533.218314538 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2555581804032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56299300015616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63587068 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37140668 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56299300015616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56299048414720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15514361 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2548100714803 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53743466610688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2547871744751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37166874 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50546814986240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63613276 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53094915701043 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53743466610688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665|J@¾Ñ –ÐM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53743466610688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62555061751296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63587068 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37166874 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2555815401984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2555815401984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65142512544768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2587450793472 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26446402 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53094669691631 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15514363 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitŒß¤eor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25571984 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65142795717632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50546797946880 _step‚113 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53743484613632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53743484613632 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63613276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26446400 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10057621 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62555085691392 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232768 trainer/global_step‚1399 % _timestamp‚1.7883964696576245e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26446402 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10056956 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2555581804032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26446400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37140668 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25571317 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 _runtime‚1533.218314538 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2555581804032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56299300015616 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63587068 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37140668 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56299300015616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56299048414720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15514361 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2548100714803 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53743466610688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2547871744751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37166874 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50546814986240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63613276 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53094915701043 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307q±ps2—- _wandbruntime‚1533  _runtime‚1533áœãxS˜Í 2loss/SRqqt2/assignment_loss‚2.9885109421135723 2loss/SRqqt3/assignment_loss‚2.8755112171173094 .loss/FBt4/detection_loss‚2.569782149578844 $loss/total_loss‚64.1914992512612 epoch‚1 1loss/SRqqt4/assignment_loss‚2.634519640310311 /loss/FRt2/detection_loss‚2.2159922122955322 1loss/SRqqt3/detection_loss‚2.4428180643490385 .loss/FBt2/detection_loss‚1.036278422466903 /loss/FRt2/assignment_loss‚5.193133688809579 /loss/FRt1/detection_loss‚2.5345028400421143 .loss/FBt3/detection_loss‚5.048183832849775 _runtime‚1533.219427816 0loss/FBt2/assignment_loss‚0.7914845542577298 .loss/FRt4/detection_loss‚2.204991228175613 _step‚114 /loss/FRt3/assignment_loss‚5.963809871673584 2loss/SRqqt1/assignment_loss‚3.4947080245384803 1loss/FBt4/assignment_loss‚0.36050927909361363 1loss/SRqqt1/detection_loss‚3.4279714203797855 1loss/SRqqt2/detection_loss‚1.9167620091010695 .loss/FBt1/assignment_loss‚0.53761960638605 0loss/FBt3/assignment_loss‚0.5833039934721975 .loss/FRt3/detection_loss‚2.275319981304082 1loss/SRqqt4/detection_loss‚1.1875797490370863 /loss/FRt1/assignment_loss‚6.375342400868734 .loss/FBt1/detection_loss‚1.640002585826594 % _timestamp‚1.7883964696597328e+09 0loss/FRt4/assignment_loss‚3.8928615372135953 trainer/global_step‚1399TW™Ñ .loss/FBt1/detection_loss‚1.640002585826594 % _timestamp‚1.7883964696597328e+09 0loss/FRt4/assignment_loss‚3.8928615372135953 trainer/global_step‚1399 2loss/SRqqt2/assignment_loss‚2.9885109421135723 2loss/SRqqt3/assignment_loss‚2.8755112171173094 .loss/FBt4/detection_loss‚2.569782149578844 $loss/total_loss‚64.1914992512612 epoch‚1 1loss/SRqqt4/assignment_loss‚2.634519640310311 /loss/FRt2/detection_loss‚2.2159922122955322 1loss/SRqqt3/detection_loss‚2.4428180643490385 .loss/FBt2/detection_loss‚1.036278422466903 /loss/FRt2/assignment_loss‚5.193133688809579 /loss/FRt1/detection_loss‚2.5345028400421143 .loss/FBt3/detection_loss‚5.048183832849775 _runtime‚1533.219427816 0loss/FBt2/assignment_loss‚0.7914845542577298 .loss/FRt4/detection_loss‚2.204991228175613 _step‚114 /loss/FRt3/assignment_loss‚5.963809871673584 2loss/SRqqt1/assignment_loss‚3.4947080245384803 1loss/FBt4/assignment_loss‚0.36050927909361363 1loss/SRqqt1/detection_loss‚3.4279714203797855 1loss/SRqqt2/detection_loss‚1.9167620091010695 .loss/FBt1/assignment_loss‚0.53761960638605 0loss/FBt3/assignment_loss‚0.5833039934721975 .loss/FRt3/detection_loss‚2.275319981304082 1loss/SRqqt4/detection_loss‚1.1875797490370863 /loss/FRt1/assignment_loss‚6.375342400868734rqÐ#wŽš‚(:ƒ Á‡ãÔšî÷Ò gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚75.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mCåC›‚(:¸ Á‡ãÔŒýÇÖ( disk./.usagePercent‚72.6294913244986 disk.rbd2.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.397093772888184 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.48828125 disk.sda1.out‚0% proc.memory.availableMB‚ 326.09375 disk.drbd1777.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚25344.48974609375 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 1618.48046875& disk.nvme0n1.out‚25344.48974609375 cpu‚25.251023690001777 proc.cpu.threads‚17 disk.sda3.out‚ 1618.48046875 network.sent‚ 2.260874e+06% disk./.usageGB‚1082.2319068908691 disk.rbd3.in‚0 disk.rbd1.out‚ 0.67626953125" disk.nvme0c0n1.in‚ 334.48828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 334.48828125# disk.dm-0.out‚25344.48974609375 disk.sda.out‚ 1618.48046875# proc.memory.rssMB‚ 2850.33984375 network.recv‚582452 disk.rbd2.out‚68.65625 disk.sda.in‚ 0.07421875$ memory_percent‚98.00968170166016:AÝgœ‚(:… ЇãÔ¬®íÒ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y"PÀE‚(:º ЇãԷмÕ$ disk./.usageGB‚1082.231876373291 disk.sda3.in‚ 0.07421875& disk.nvme0n1.out‚25344.90673828125# memory_percent‚98.6215353012085 network.recv‚588856( proc.memory.availableMB‚ 225.84765625 cpu‚25.249159365363543 network.sent‚ 2.369072e+06 disk.sda3.out‚ 1637.5234375 disk.rbd1.in‚0 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚25344.90673828125 disk.rbd0.in‚0 disk.dm-0.in‚ 334.48828125 disk.rbd3.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚72.62948927643833" disk.nvme0c0n1.in‚ 334.48828125 disk.dm-1.out‚ 1637.5234375 disk.sda1.out‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2850.33984375 disk.dm-1.in‚ 0.07421875# disk.dm-0.out‚25344.90673828125 disk.sda.out‚ 1637.5234375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 334.48828125 disk.rbd2.out‚ 68.66796875 disk.rbd3.in‚0* proc.memory.percent‚17.397093772888184 disk.rbd1.out‚ 0.67626953125 disk.sda.in‚ 0.07421875 disk.sda2.out‚02ëY›ž‚(: ߇ãÔ›¨öÒ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚89.301' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.860200000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Kø76Ÿ‚(:« ߇ãÔÍØŒÖ proc.memory.rssMB‚ 2850.34375 disk.dm-0.in‚ 334.48828125 disk.rbd3.in‚0 disk.rbd1.out‚ 0.67626953125 disk.dm-1.in‚ 0.07421875$ memory_percent‚98.66247177124023 disk.rbd0.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 334.48828125 cpu‚25.247429123981505" disk.nvme0n1.out‚ 25444.4765625 disk.dm-0.out‚ 25444.4765625 disk.sda.out‚ 1665.8828125 disk.rbd2.in‚0 disk.rbd2.out‚ 68.73046875 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 25444.4765625* proc.memory.percent‚17.397117614746094 disk.sda1.out‚0 network.sent‚ 2.374215e+06 network.recv‚592559 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 1665.8828125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 1665.8828125) disk./.usagePercent‚72.62949183651367% disk./.usageGB‚1082.2319145202637 disk.nvme0n1.in‚ 334.48828125 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0& proc.memory.availableMB‚ 219.140625 disk.rbd0.out‚02ÈóË2 - _wandbruntime‚1585  _runtime‚1585Ô, È¡Â $ _timestamp‚1.788396521943503e+09 _runtime‚1585.503339118 _step‚115 ' lr-AdamW/pg1‚0.00022274644662716563 ' lr-AdamW/pg2‚0.00022274644662716563 trainer/global_step‚1449’mÌ¢Æ ' lr-AdamW/pg1‚0.00022274644662716563 ' lr-AdamW/pg2‚0.00022274644662716563 trainer/global_step‚1449 $ _timestamp‚1.788396521943503e+09 _runtime‚1585.503339118 _step‚115sF¿j2£- _wandbruntime‚1585  _runtime‚1585 øó#È'¤ÂO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 _step‚116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 _runtime‚1585.504358589 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65785048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38421695 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27337173 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27337171 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38447875 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2644478074880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2636268274519 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65758866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16036360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚738 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2677003986432 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54894720206027 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58207683307008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55563456819712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283186176 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55563438816768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2644478074880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10396624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 trainer/global_step‚1449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55563456819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16036358 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52258205922304 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 % _timestamp‚1.7883965219443052e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52258222961664Û¸v¼x¥ÆO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2644244490240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55563438816768 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58207683307008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27337173 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26432984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚740 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259246080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58207683307008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55563456819712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283186176 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55563438816768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2644478074880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10396624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 trainer/global_step‚1449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55563456819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16036358 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52258205922304 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 % _timestamp‚1.7883965219443052e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52258222961664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 _step‚116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 _runtime‚1585.504358589 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65785048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38421695 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27337173 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27337171 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38447875 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2644478074880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ZHDevicI•MbT eStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2636268274519 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65758866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16036360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚738 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2677003986432 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54894720206027 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 _runtime‚1586.495829434 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27355350 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38447770 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38473976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 % _timestamp‚1.7883965229351802e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2646287236608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58246886346240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2638072007959 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55600599109632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27355350 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67402833036800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10403492 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 trainer/global_step‚1449 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _step‚117 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65803120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64724031176192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2679093422080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54931450633051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291561472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27355352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16047011 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2638300978011 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65829328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2646053638656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2646053638656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65803120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267621376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58246634745344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26449831 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55600581106688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65829328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26450505 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67403124598272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2646287236608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64724007236096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2678825800704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58246886346240 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55600599109632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27355352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38473976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55600581106688 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2646053638656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2646053638656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65803120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267621376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58246634745344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26449831 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55600581106688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65829328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26450505 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67403124598272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2646287236608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64724007236096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2678825800704 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58246886346240 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55600599109632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27355352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38473976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55600581106688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 _runtime‚1586.495829434 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27355350 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38447770 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38473976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 % _timestamp‚1.7883965229351802e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2646287236608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58246886346240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2638072007959 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55600599109632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27355350 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67402833036800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10403492 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 trainer/global_step‚1449 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _step‚117 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65803120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64724031176192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2679093422080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54931450633051uO–ìÓ2©- _wandbruntime‚1586  _runtime‚1586€·MéWªÑ trainer/global_step‚1449 _runtime‚1586.497446341 0loss/FRt3/assignment_loss‚5.9770277081703655 /loss/FRt4/assignment_loss‚3.766305130460988 /loss/FBt1/detection_loss‚1.2402756432524047 /loss/FBt2/detection_loss‚2.1588179622070456 0loss/FRt1/assignment_loss‚5.8895869016647335 2loss/SRqqt4/assignment_loss‚2.6842345056995267 /loss/FBt1/assignment_loss‚0.326679458623685 0loss/FBt2/assignment_loss‚0.8299679606061545 /loss/FRt4/detection_loss‚2.2757106371547864 /loss/FBt4/detection_loss‚2.7270774176487556 $ _timestamp‚1.788396522937721e+09 2loss/SRqqt2/assignment_loss‚2.9138038837247424 1loss/SRqqt3/assignment_loss‚2.827824080431903 0loss/FBt4/assignment_loss‚0.4081316205733506 epoch‚1 /loss/FRt3/detection_loss‚2.3137837243931636 1loss/SRqqt4/detection_loss‚0.9090260439502296 2loss/SRqqt1/assignment_loss‚2.3956045670942827 /loss/FBt3/assignment_loss‚0.539058908904818 0loss/SRqqt1/detection_loss‚4.374779571186412 %loss/total_loss‚63.27121406642352 /loss/FRt1/detection_loss‚3.2089492589235307 .loss/FBt3/detection_loss‚3.817767723040147 /loss/FRt2/detection_loss‚2.1951073463316315 /loss/FRt2/assignment_loss‚5.045478745743081 1loss/SRqqt3/detection_loss‚3.0372741001623647 _step‚118 1loss/SRqqt2/detection_loss‚1.4089411664754152c%*<[«Õ 0loss/FRt3/assignment_loss‚5.9770277081703655 /loss/FRt4/assignment_loss‚3.766305130460988 /loss/FBt1/detection_loss‚1.2402756432524047 /loss/FBt2/detection_loss‚2.1588179622070456 0loss/FRt1/assignment_loss‚5.8895869016647335 2loss/SRqqt4/assignment_loss‚2.6842345056995267 /loss/FBt1/assignment_loss‚0.326679458623685 0loss/FBt2/assignment_loss‚0.8299679606061545 /loss/FRt4/detection_loss‚2.2757106371547864 /loss/FBt4/detection_loss‚2.7270774176487556 $ _timestamp‚1.788396522937721e+09 2loss/SRqqt2/assignment_loss‚2.9138038837247424 1loss/SRqqt3/assignment_loss‚2.827824080431903 0loss/FBt4/assignment_loss‚0.4081316205733506 epoch‚1 /loss/FRt3/detection_loss‚2.3137837243931636 1loss/SRqqt4/detection_loss‚0.9090260439502296 2loss/SRqqt1/assignment_loss‚2.3956045670942827 /loss/FBt3/assignment_loss‚0.539058908904818 0loss/SRqqt1/detection_loss‚4.374779571186412 %loss/total_loss‚63.27121406642352 /loss/FRt1/detection_loss‚3.2089492589235307 .loss/FBt3/detection_loss‚3.817767723040147 /loss/FRt2/detection_loss‚2.1951073463316315 /loss/FRt2/assignment_loss‚5.045478745743081 1loss/SRqqt3/detection_loss‚3.0372741001623647 _step‚118 1loss/SRqqt2/detection_loss‚1.4089411664754152 trainer/global_step‚1449 _runtime‚1586.497446341vUo²Ž¬‚(:ƒ î‡ãÔŒŽ”Ó gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚75.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯‚(:³ ý‡ãÔ‘÷ÉÖ disk.dm-1.in‚ 0.07421875! disk.dm-0.out‚25543.494140625* proc.memory.percent‚17.397117614746094 cpu‚25.24408982026938 proc.cpu.threads‚17) disk./.usagePercent‚72.62952537350066 disk.sda3.in‚ 0.07421875 disk.drbd1777.out‚0 disk.rbd2.in‚0 network.recv‚602711 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚25543.494140625 disk.rbd0.out‚0 disk.dm-0.in‚ 334.48828125$ memory_percent‚98.66423606872559' proc.memory.availableMB‚ 218.8515625% disk./.usageGB‚1082.2324142456055 disk.sda.out‚ 1719.02734375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.67626953125" disk.nvme0c0n1.in‚ 334.48828125 disk.rbd0.in‚0 disk.sda.in‚ 0.07421875 proc.memory.rssMB‚ 2850.34375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 network.sent‚ 2.487873e+06 disk.sda3.out‚ 1719.02734375 disk.dm-1.out‚ 1719.02734375 disk.nvme0n1.in‚ 334.48828125$ disk.nvme0n1.out‚25543.494140625 disk.rbd2.out‚ 70.3203125xð‚(:… ŒˆãÔ¤ŽäÔ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚88.772' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7544 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£¾ùX@±‚(:µ ŒˆãÔ¦²§× disk.rbd2.out‚ 70.3203125 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚25641.8857421875( proc.memory.availableMB‚ 219.54296875 disk.sda1.out‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 1720.3515625 disk.sda.in‚ 0.07421875) disk./.usagePercent‚72.62952153338765 disk.rbd1.out‚ 0.67626953125" disk.nvme0c0n1.in‚ 334.48828125' disk.nvme0c0n1.out‚25641.8857421875 disk.nvme0n1.in‚ 334.48828125" disk.dm-0.out‚25641.8857421875* proc.memory.percent‚17.397117614746094% disk./.usageGB‚1082.2323570251465 disk.rbd0.in‚0 disk.dm-0.in‚ 334.48828125 disk.sda.out‚ 1720.3515625$ memory_percent‚98.66001605987549 disk.sda2.out‚0 network.sent‚ 2.494407e+06 network.recv‚607624 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 1720.3515625 cpu‚25.242298803600924 disk.drbd1777.in‚0 disk.rbd3.in‚0 proc.memory.rssMB‚ 2850.34375›@§T²‚(:… ›ˆãÔÞìŒÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.829' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5658 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò¹´Û@³‚(:µ ›ˆãÔÿýØÖ% disk./.usageGB‚1082.2324142456055 disk.rbd0.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 0.67626953125 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 1738.4765625" disk.nvme0c0n1.in‚ 334.48828125& disk.nvme0c0n1.out‚25642.681640625 disk.rbd0.in‚0 disk.sda.out‚ 1738.4765625 disk.drbd1777.out‚0 disk.rbd2.out‚70.43017578125( proc.memory.availableMB‚ 219.29296875$ disk.nvme0n1.out‚25642.681640625 disk.sda2.in‚0 disk.sda3.out‚ 1738.4765625 network.recv‚611234) disk./.usagePercent‚72.62952537350066 disk.drbd1777.in‚0 disk.sda1.out‚0 network.sent‚ 2.49944e+06 disk.dm-0.in‚ 334.48828125! disk.dm-0.out‚25642.681640625 disk.sda2.out‚0$ memory_percent‚98.66154193878174 proc.cpu.threads‚17 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2850.34375 disk.sda1.in‚0 disk.nvme0n1.in‚ 334.48828125* proc.memory.percent‚17.397117614746094 cpu‚25.240582630458324—™yM2´- _wandbruntime‚1638  _runtime‚1638M ɵà ' lr-AdamW/pg1‚0.00022262340483727698 ' lr-AdamW/pg2‚0.00022262340483727698 trainer/global_step‚1499 % _timestamp‚1.7883965748849537e+09 _runtime‚1638.444943576 _step‚119Gù‡Í¶Ç _step‚119 ' lr-AdamW/pg1‚0.00022262340483727698 ' lr-AdamW/pg2‚0.00022262340483727698 trainer/global_step‚1499 % _timestamp‚1.7883965748849537e+09 _runtime‚1638.444943576wãîà2·- _wandbruntime‚1638  _runtime‚1638ÆU 4¸ÁO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10740046 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66849573811200 $ _timestamp‚1.788396574886012e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57420553312768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2734716324864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2726468537727 _step‚120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16569008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10739333 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 _runtime‚1638.446428856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60155269637632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56731255138035 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39728797 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2768698920960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60155521225216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54004557630464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60155269637632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57420553312768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28246123 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39754977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39754977 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39728797 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57420553312768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2734716324864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2726468537727 _step‚120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16569008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10739333 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 _runtime‚1638.446428856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60155269637632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56731255138035 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39728797 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2768698920960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60155521225216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54004557630464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60155269637632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57420553312768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28246123 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39754977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39754977 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39728797 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28264300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57457695602688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2770525571584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 _runtime‚1639.467570235 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39781078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39781078 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27325826 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10746165 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68019172 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68019172 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56767739555647 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68045380 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39754872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57457695602688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27326527 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56767985565059 _step‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2736525473280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2728272271167 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16579663 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2736759071232 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60194472676864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60194221075968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2736525473280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54039484323840 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54039467284480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68045380 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2728501241219 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28264302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1é"XM{ ¼ÐM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39754872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16579661 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28264300 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66892952720896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68045380 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMoni¹ŽË[tor.on_train_batch_end/active.small_pool.freed‚39754872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57457695602688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27326527 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56767985565059 _step‚121 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2736525473280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2728272271167 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16579663 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2736759071232 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60194472676864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60194221075968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2736525473280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54039484323840 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54039467284480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68045380 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2728501241219 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28264302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7883965759068546e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28264300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57457695602688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2770525571584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 _runtime‚1639.467570235 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39781078 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39781078 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27325826 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10746165 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68019172 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68019172 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56767739555647 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62102855968256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2825421744128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59277667808768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62103107555840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17101660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17101658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11082988 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70217152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55750875259904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70190970 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29155073 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41062079 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41035899 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2825421744128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69018543236096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59277685811712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29155071 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11083684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58567790070043 _runtime‚1691.205139772 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 _step‚124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71878450433024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2816668800935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62102855968256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55750892299264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2825188159488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29155071 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253ìeK} ËÆO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29155071 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11083684 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58567790070043 _runtime‚1691.205139772 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 _step‚124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71878450433024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2816668800935 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62102855968256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55750892299264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2825188159488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29155071 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59277667808768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281089024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62102855968256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2825421744128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59277667808768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62103107555840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17101660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17101658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11082988 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70217152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55750875259904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70190970 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29155073 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41062079 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41035899 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2825421744128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69018543236096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 WJDeviceStatsMonitor.on_trai]ÉÝ@Æn_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59277685811712|ÙDZ2Ì- _wandbruntime‚1692  _runtime‚1692Ȫ0Ò&ÍÌM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69061922145792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29173250 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70261432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41088180 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58604274487655 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59314828101632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28202379 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69061898205696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41088180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59314828101632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2862025225728 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29173252 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29173250 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55785801953280 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59314810098688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28203064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70235224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7883966286479292e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62142059007488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29173252 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287367168 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71923660004352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2861761798656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59314810098688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58604520497067 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55785818992640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 _step‚125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11090068 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11090751 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263427072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62142059007488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17112311 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71923947371520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17112313 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2818701504427 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2818472534375 _runtime‚1692.208473576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70261432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2826997307904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41061974 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1549›RO»Ö&ÎÐM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70261432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2826997307904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41061974 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1549 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69061922145792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29173250 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70261432 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41088180 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58604274487655 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59314828101632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28202379 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69061898205696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41088180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59314828101632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2862025225728 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29173252 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29173250 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55785801953280 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59314810098688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28203064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70235224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7883966286479292e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62142059007488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29173252 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287367168 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71923660004352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2861761798656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59314810098688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58604520497067 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55785818992640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 _step‚125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11090068 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11090751 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263427072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62142059007488 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17112311 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71923947371520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17112313 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2818701504427 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2818472534375 _runtime‚1692.208473576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0}­3ÃW2Ï- _wandbruntime‚1692  _runtime‚1692È¥-JTÐÎ %loss/total_loss‚66.63927350349348 _runtime‚1692.209755973 $ _timestamp‚1.788396628649917e+09 0loss/FRt4/assignment_loss‚2.7506896815401443 0loss/FBt4/assignment_loss‚0.3075044393894031 1loss/SRqqt4/detection_loss‚1.0765781240101824 /loss/FBt4/detection_loss‚1.9927894688314862 1loss/SRqqt2/detection_loss‚1.7489127240260132 epoch‚1 0loss/FBt2/assignment_loss‚0.5913483425974846 1loss/SRqqt1/detection_loss‚3.2508456707000732 trainer/global_step‚1549 .loss/FRt1/assignment_loss‚6.60588738792821 2loss/SRqqt1/assignment_loss‚2.9884721264243126 1loss/SRqqt3/assignment_loss‚2.816241428769868 1loss/SRqqt4/assignment_loss‚2.507416881145315 0loss/FBt1/assignment_loss‚0.3497970967618846 .loss/FRt1/detection_loss‚2.896284636698271 /loss/FRt3/detection_loss‚2.5290713172405956 0loss/FRt3/assignment_loss‚6.0053310500250925 /loss/FRt4/detection_loss‚2.1259678094945054 1loss/SRqqt3/detection_loss‚2.8438814200203995 /loss/FBt3/detection_loss‚3.3449505482401167 _step‚126 0loss/FBt3/assignment_loss‚0.6354176825839204 .loss/FRt2/detection_loss‚2.340234960913658 .loss/FBt1/detection_loss‚1.875572263311559 /loss/FRt2/assignment_loss‚4.308053586483002 2loss/SRqqt2/assignment_loss‚2.7484814282506704 -loss/FBt2/detection_loss‚7.99954342810731Yo°ÒXÑÒ %loss/total_loss‚66.63927350349348 _runtime‚1692.209755973 $ _timestamp‚1.788396628649917e+09 0loss/FRt4/assignment_loss‚2.7506896815401443 0loss/FBt4/assignment_loss‚0.3075044393894031 1loss/SRqqt4/detection_loss‚1.0765781240101824 /loss/FBt4/detection_loss‚1.9927894688314862 1loss/SRqqt2/detection_loss‚1.7489127240260132 epoch‚1 0loss/FBt2/assignment_loss‚0.5913483425974846 1loss/SRqqt1/detection_loss‚3.2508456707000732 trainer/global_step‚1549 .loss/FRt1/assignment_loss‚6.60588738792821 2loss/SRqqt1/assignment_loss‚2.9884721264243126 1loss/SRqqt3/assignment_loss‚2.816241428769868 1loss/SRqqt4/assignment_loss‚2.507416881145315 0loss/FBt1/assignment_loss‚0.3497970967618846 .loss/FRt1/detection_loss‚2.896284636698271 /loss/FRt3/detection_loss‚2.5290713172405956 0loss/FRt3/assignment_loss‚6.0053310500250925 /loss/FRt4/detection_loss‚2.1259678094945054 1loss/SRqqt3/detection_loss‚2.8438814200203995 /loss/FBt3/detection_loss‚3.3449505482401167 _step‚126 0loss/FBt3/assignment_loss‚0.6354176825839204 .loss/FRt2/detection_loss‚2.340234960913658 .loss/FBt1/detection_loss‚1.875572263311559 /loss/FRt2/assignment_loss‚4.308053586483002 2loss/SRqqt2/assignment_loss‚2.7484814282506704 -loss/FBt2/detection_loss‚7.99954342810731~ŠæÒ‚(:… ׈ãÔÃßúÒ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6354 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N/DKÓ‚(:À ׈ãÔäòÄÕ network.recv‚629846 disk.dm-1.out‚ 1842.37890625 disk.rbd0.out‚0# disk.dm-0.out‚25938.59521484375 disk.sda.in‚ 0.07421875 disk.sda.out‚ 1842.37890625( proc.memory.availableMB‚ 219.17578125 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 334.48828125) disk./.usagePercent‚72.62954047794517 disk.sda2.in‚0( disk.nvme0c0n1.out‚25938.59521484375 disk.dm-0.in‚ 334.48828125$ memory_percent‚98.66225719451904 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚72.16064453125 disk.rbd3.out‚0 disk.sda3.out‚ 1842.37890625& disk.nvme0n1.out‚25938.59521484375# proc.memory.rssMB‚ 2850.34765625 network.sent‚ 2.6246e+06 disk.rbd1.out‚ 0.67626953125 disk.nvme0n1.in‚ 334.48828125* proc.memory.percent‚17.397141456604004 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.in‚0% disk./.usageGB‚1082.2326393127441 disk.rbd1.in‚0 cpu‚25.234265686952472û¡Àá›Ô‚(: æˆãÔñÙçÒ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.245' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.049000000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`}¼'IÕ‚(:¾ æˆãÔ†Ú¤Ö network.sent‚ 2.733201e+06 disk.dm-1.out‚ 1877.12109375% disk./.usageGB‚1081.0140228271484 disk.rbd2.out‚72.16064453125 disk.rbd3.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚26037.31396484375 disk.dm-0.in‚ 334.48828125 disk.sda1.in‚0) disk./.usagePercent‚72.54775810310952# disk.dm-0.out‚26037.31396484375" memory_percent‚98.659348487854# proc.memory.rssMB‚ 2850.34765625* proc.memory.percent‚17.397141456604004( disk.nvme0c0n1.out‚26037.31396484375 disk.dm-1.in‚ 0.07421875 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 334.48828125 cpu‚25.2329398600848 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 1877.12109375 disk.nvme0n1.in‚ 334.48828125 disk.sda.out‚ 1877.12109375 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 219.65234375 network.recv‚636483 disk.rbd1.out‚ 0.67626953125Ôý›Ö‚(: õˆãÔúäÓ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.243' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.848599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Œ~Eׂ(:º õˆãÔ¼†áÖ( disk./.usagePercent‚72.5477565670643 disk.nvme0n1.in‚ 334.48828125 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.rbd1.out‚ 0.76220703125* proc.memory.percent‚17.397141456604004 network.sent‚ 2.738622e+06 disk.sda.out‚ 1879.3515625 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 219.16015625% disk./.usageGB‚1081.0139999389648' disk.nvme0c0n1.out‚26037.7744140625 disk.rbd0.in‚0 cpu‚25.231358824881564" disk.nvme0c0n1.in‚ 334.48828125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 1879.3515625% disk.nvme0n1.out‚26037.7744140625 disk.dm-1.in‚ 0.07421875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 network.recv‚640271 disk.dm-0.in‚ 334.48828125 disk.rbd2.out‚ 72.3291015625 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2850.34765625 disk.dm-1.out‚ 1879.3515625 disk.rbd0.out‚0 proc.cpu.threads‚17" disk.dm-0.out‚26037.7744140625 disk.sda1.in‚0 disk.sda3.in‚ 0.07421875$ memory_percent‚98.66235256195068~Ø‚(:… „‰ãÔ¼»¼Ó gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚81.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.3718 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W¥­°HÙ‚(:½ „‰ãÔ´ƒ­Ö' disk.nvme0c0n1.out‚26038.0986328125% disk.nvme0n1.out‚26038.0986328125 disk.sda1.in‚0$ memory_percent‚98.66087436676025# proc.memory.rssMB‚ 2850.34765625* proc.memory.percent‚17.397141456604004 disk.sda.in‚ 0.07421875 disk.sda.out‚ 1880.01171875 disk.rbd2.in‚0 network.sent‚ 2.744175e+06( disk./.usagePercent‚72.5477527269513% disk./.usageGB‚1081.0139427185059 disk.sda2.in‚0 disk.sda3.out‚ 1880.01171875 disk.dm-1.in‚ 0.07421875 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 334.48828125 disk.sda1.out‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.48828125" disk.dm-0.out‚26038.0986328125 disk.rbd1.out‚ 0.82470703125" disk.nvme0c0n1.in‚ 334.48828125 disk.rbd2.out‚ 72.3291015625( proc.memory.availableMB‚ 219.40234375 network.recv‚644092 disk.rbd3.in‚0 disk.dm-1.out‚ 1880.01171875 cpu‚25.229972098567362¥ˆB2Ú- _wandbruntime‚1742  _runtime‚1742cR&ØÆÛÀ % _timestamp‚1.7883966793526244e+09 _runtime‚ 1742.91243814 _step‚127 & lr-AdamW/pg1‚0.0002223078822623192 & lr-AdamW/pg2‚0.0002223078822623192 trainer/global_step‚1599°fN*ÊÜÄ & lr-AdamW/pg2‚0.0002223078822623192 trainer/global_step‚1599 % _timestamp‚1.7883966793526244e+09 _runtime‚ 1742.91243814 _step‚127 & lr-AdamW/pg1‚0.0002223078822623192^I(2Ý- _wandbruntime‚1742  _runtime‚1742¯FÈ1‹ÞÂO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2915893578752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2915659994112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60404078992847 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64050693886464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30064023 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2907098033987 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17634308 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 _LDeviceStatsMonitor.on_train_batchä }„="_start/requested_bytes.large_pool.allocated‚57497226968064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚742 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61134782304768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74139103729664 _runtime‚1742.913418678 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72433204 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 261343232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42369181 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30064021 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚740 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 285283328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74138818446336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17634310 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42343001 trainer/global_step‚1599 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64050442298880 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61134782304768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2951615008768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2951353665536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64050442298880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2915659994112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71187488720896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61134800307712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29062126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11427076 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2906869064143óG®ÛÍ'ßÇO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72407022 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64050693886464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2915893578752 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64050442298880 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61134782304768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2951615008768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2951353665536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64050442298880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2915659994112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71187488720896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61134800307712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29062126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11427076 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2906869064143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2915893578752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2915659994112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60404078992847 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64050693886464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30064023 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2907098033987 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17634308 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57497226968064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚742 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61134782304768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74139103729664 _runtime‚1742.913418678 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72433204 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 261343232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42369181 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30064021 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚740 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 285283328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74138818446336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17634310 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42343001 trainer/global_step‚1599 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72451276 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61171942597632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61171942597632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2917702740480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29079051 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60440809419663 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42395282 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17644961 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42369076 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2953441913344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42369076 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64089393737216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61171924594688 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 _step‚129 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74184020079616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17644963 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71230843690496 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2908672797583 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42395282 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72477484 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2917469142528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2917702740480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚735 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11434090 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60441055429075 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72477484 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74184309543936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2917469142528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚733 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7883966803171642e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61171924594688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11434823 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57532136622080 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72451276 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57532153661440íË)ÏâÑM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61171924594688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11434823 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57532136622080 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72451276 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57532153661440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30082200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2908901767635 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30082202 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30082202 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72451276 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61171942597632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61171942597632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2917702740480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29079051 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚60440809419663 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42395282 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17644961 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_eEÒQnd/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42369076 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2953441913344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42369076 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64089393737216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61171924594688 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 _step‚129 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74184020079616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17644963 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71230843690496 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2908672797583 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42395282 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72477484 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2917469142528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2917702740480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚735 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11434090 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60441055429075 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72477484 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74184309543936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2917469142528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚733 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7883966803171642e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560¾£,U2ã- _wandbruntime‚1743  _runtime‚1743”•íÕWäÑ /loss/FBt3/assignment_loss‚0.385802570451051 1loss/FBt4/assignment_loss‚0.33027678417184214 /loss/FBt2/detection_loss‚7.8597510814666744 /loss/FRt2/assignment_loss‚4.665891162554423 2loss/SRqqt3/assignment_loss‚2.3527209134328935 0loss/FBt1/assignment_loss‚0.5262388645484614 /loss/FRt2/detection_loss‚2.0628272677461306 /loss/FRt3/detection_loss‚2.4238600609641447 _runtime‚1743.879311044 $ _timestamp‚1.788396680319583e+09 /loss/FRt1/assignment_loss‚6.586146074182847 0loss/FBt2/assignment_loss‚0.8098766922950744 2loss/SRqqt2/assignment_loss‚2.5417263388633726 /loss/FBt1/detection_loss‚1.1577448968878963 trainer/global_step‚1599 /loss/FRt3/assignment_loss‚6.023464651668773 1loss/SRqqt1/detection_loss‚3.2566113471984863 1loss/SRqqt4/detection_loss‚1.3935256572679067 0loss/FRt4/assignment_loss‚3.0291523039340973 .loss/FRt1/detection_loss‚2.936865680357989 2loss/SRqqt4/assignment_loss‚2.6693606667029552 1loss/SRqqt2/detection_loss‚1.5947758809264216 1loss/SRqqt3/detection_loss‚3.1827327523912703 .loss/FBt3/detection_loss‚3.713727021217346 %loss/total_loss‚66.62399686178148 epoch‚1 _step‚130 2loss/SRqqt1/assignment_loss‚3.3133155247744392 .loss/FRt4/detection_loss‚2.068559836458277 .loss/FBt4/detection_loss‚1.739042831318719¤ÑLb\åÖ 2loss/SRqqt4/assignment_loss‚2.6693606667029552 1loss/SRqqt2/detection_loss‚1.5947758809264216 1loss/SRqqt3/detection_loss‚3.1827327523912703 .loss/FBt3/detection_loss‚3.713727021217346 %loss/total_loss‚66.62399686178148 epoch‚1 _step‚130 2loss/SRqqt1/assignment_loss‚3.3133155247744392 .loss/FRt4/detection_loss‚2.068559836458277 .loss/FBt4/detection_loss‚1.739042831318719 /loss/FBt3/assignment_loss‚0.385802570451051 1loss/FBt4/assignment_loss‚0.33027678417184214 /loss/FBt2/detection_loss‚7.8597510814666744 /loss/FRt2/assignment_loss‚4.665891162554423 2loss/SRqqt3/assignment_loss‚2.3527209134328935 0loss/FBt1/assignment_loss‚0.5262388645484614 /loss/FRt2/detection_loss‚2.0628272677461306 /loss/FRt3/detection_loss‚2.4238600609641447 _runtime‚1743.879311044 $ _timestamp‚1.788396680319583e+09 /loss/FRt1/assignment_loss‚6.586146074182847 0loss/FBt2/assignment_loss‚0.8098766922950744 2loss/SRqqt2/assignment_loss‚2.5417263388633726 /loss/FBt1/detection_loss‚1.1577448968878963 trainer/global_step‚1599 /loss/FRt3/assignment_loss‚6.023464651668773 1loss/SRqqt1/detection_loss‚3.2566113471984863 1loss/SRqqt4/detection_loss‚1.3935256572679067 0loss/FRt4/assignment_loss‚3.0291523039340973 .loss/FRt1/detection_loss‚2.936865680357989‚žÖâæ‚(:… “‰ãÔˆ©ûÓ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.133' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8266 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 EoEç‚(:º “‰ãÔÚœÈ× disk.sda2.in‚0 disk.dm-1.out‚ 1881.42578125 network.recv‚647598 disk.sda3.out‚ 1881.42578125( proc.memory.availableMB‚ 219.08203125 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 334.48828125! disk.dm-0.out‚26305.427734375* proc.memory.percent‚17.397141456604004 network.sent‚ 2.749226e+06) disk./.usagePercent‚72.56521090474128 disk.rbd3.in‚0 disk.rbd1.out‚ 0.82470703125 disk.rbd0.in‚0 disk.sda.in‚ 0.07421875# proc.memory.rssMB‚ 2850.34765625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 334.48828125 disk.rbd2.out‚ 73.9814453125$ memory_percent‚98.66282939910889 disk.nvme0n1.in‚ 334.48828125 disk.drbd1777.out‚0 cpu‚25.228421380155687 disk.rbd3.out‚0& disk.nvme0c0n1.out‚26305.427734375 disk.rbd0.out‚0$ disk.nvme0n1.out‚26305.427734375 disk.sda.out‚ 1881.42578125 disk.sda1.out‚0 disk.rbd2.in‚0$ disk./.usageGB‚1081.274082183838äbà†è‚(:… ¢‰ãÔµ¬ÊÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3862 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùOeã?é‚(:´ ¢‰ãÔ쇫Ö disk.dm-1.in‚ 0.07421875( proc.memory.availableMB‚ 219.31640625 proc.cpu.threads‚17% disk./.usageGB‚1081.2740097045898 disk.rbd0.out‚0* proc.memory.percent‚17.397141456604004 cpu‚25.22696739402464 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 1901.078125 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875& disk.nvme0c0n1.out‚26305.755859375 disk.nvme0n1.in‚ 334.48828125$ disk.nvme0n1.out‚26305.755859375 disk.sda1.out‚0 disk.rbd1.out‚ 0.85693359375 disk.dm-0.in‚ 334.48828125! disk.dm-0.out‚26305.755859375 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2850.34765625) disk./.usagePercent‚72.56520604059813" disk.nvme0c0n1.in‚ 334.48828125 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 73.9814453125$ memory_percent‚98.66139888763428 disk.sda3.out‚ 1901.078125 disk.sda.out‚ 1901.078125 disk.sda1.in‚0 disk.sda2.in‚0 network.sent‚ 2.857432e+06 network.recv‚654053 disk.rbd1.in‚0HKnòê‚(:„ ±‰ãÔþ…þÓ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚86.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.299 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E0-Lë‚(:Á ±‰ãÔ‘Ø£Ö) disk./.usagePercent‚72.56520988071114 disk.rbd3.in‚0 disk.sda2.out‚0 network.sent‚ 2.862894e+06% disk./.usageGB‚1081.2740669250488 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 334.48828125 disk.dm-0.in‚ 334.48828125 disk.sda.in‚ 0.07421875 disk.rbd2.out‚74.08349609375 disk.sda3.out‚ 1921.95703125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 334.48828125 disk.sda3.in‚ 0.07421875 disk.drbd1777.out‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚26306.48388671875# proc.memory.rssMB‚ 2850.34765625 proc.cpu.threads‚17 disk.sda.out‚ 1921.95703125 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.07421875( proc.memory.availableMB‚ 219.30859375 cpu‚25.225642174742816 network.recv‚658079# disk.dm-0.out‚26306.48388671875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 1921.95703125# memory_percent‚98.6614465713501* proc.memory.percent‚17.397141456604004& disk.nvme0n1.out‚26306.48388671875 disk.sda2.in‚0 disk.rbd1.out‚ 0.85693359375 disk.rbd0.in‚0¥Çžž2ì- _wandbruntime‚1795  _runtime‚1795é¹4æÈí ' lr-AdamW/pg1‚0.00022211546723168173 ' lr-AdamW/pg2‚0.00022211546723168173 trainer/global_step‚1649 $ _timestamp‚1.788396731515716e+09 _runtime‚1795.075572499 _step‚131ò‘CÍîÇ trainer/global_step‚1649 $ _timestamp‚1.788396731515716e+09 _runtime‚1795.075572499 _step‚131 ' lr-AdamW/pg1‚0.00022211546723168173 ' lr-AdamW/pg2‚0.00022211546723168173ƒÑ°Ô2ï- _wandbruntime‚1795  _runtime‚1795HÅçÆ'ðÀO _step‚132 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265537536 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65998028629504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65998280217088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62240859934059 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30972973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43650103 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65998028629504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3006131828736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30972971 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289477632 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62991914803712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2997069327351 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65998280217088 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29941123 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76399427412480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2997298297195 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3006365413376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74649256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43676283 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3042993206784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3006131828736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11774864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62991896800768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62240613924855 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚01^‡Pe!ñÅO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65998280217088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62240859934059 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30972973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43650103 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65998028629504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3006131828736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30972971 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289477632 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62991914803712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2997069327351 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65998280217088 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29941123 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76399427412480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2997298297195 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3006365413376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74649256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43676283 cQDeviceSt㸠fatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3042993206784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3006131828736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11774864 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62991896800768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62240613924855 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚132 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265537536 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65998028629504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0„^„T2ò- _wandbruntime‚1796  _runtime‚1796×èêÑ&óËM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73399789175296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3044556037120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66037231668736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62277590361083 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788396732517815e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30991150 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3007940977152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2998873060791 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59278471290880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74667328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30991152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30991152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63029039090688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3007940977152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2999102030843 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74693536 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30991150 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43702384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63029039090688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1796.078241143 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73399813115392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74667328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3044821561344 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43702384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43676178 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59278488330240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29959141 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76444634676736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63029057093632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18177613 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11780798 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66037231668736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18177611 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3008174575104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63029057093632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29958409[ðc´Ö&ôÐM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74667328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3044821561344 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43702384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43676178 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59278488330240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29959141 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76444634676736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63029057093632 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18177613 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11780798 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66037231668736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18177611 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3008174575104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63029057093632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29958409 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73399789175296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3044556037120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66037231668736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62277590361083 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788396732517815e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30991150 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3007940977152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2998873060791 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59278471290880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74667328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30991152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30991152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63029039090688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3007940977152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2999102030843 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74693536 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30991150 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43702384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63029039090688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1796.078241143 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73399813115392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 O2õ- _wandbruntime‚1796  _runtime‚1796Æ^YöÓ 2loss/SRqqt3/assignment_loss‚2.5793152348748567 1loss/SRqqt2/detection_loss‚1.5670865545670192 .loss/FBt2/detection_loss‚20.00162625312805 .loss/FBt3/detection_loss‚3.352744793891907 % _timestamp‚1.7883967325199645e+09 0loss/FRt1/assignment_loss‚6.2488607035742865 1loss/SRqqt2/assignment_loss‚2.579511965115865 1loss/FBt1/assignment_loss‚0.39251982724848833 /loss/FRt3/detection_loss‚2.6086863511921585 trainer/global_step‚1649 0loss/FBt2/assignment_loss‚0.6412692666053772 1loss/SRqqt4/detection_loss‚1.0521723219881887 /loss/FRt3/assignment_loss‚5.987603922148009 0loss/FRt4/assignment_loss‚3.1496125058881166 1loss/FBt4/assignment_loss‚0.24224318367524492 /loss/FRt1/detection_loss‚3.0502207279205322 /loss/FRt4/detection_loss‚1.7701827770677105 _step‚134 /loss/FRt2/detection_loss‚2.2677211011094704 1loss/SRqqt3/detection_loss‚2.6858985280168466 .loss/FBt1/detection_loss‚2.277084704224225 _runtime‚ 1796.07975204 .loss/FRt2/assignment_loss‚4.75623522911753 2loss/SRqqt4/assignment_loss‚2.5738959053288335 0loss/FBt3/assignment_loss‚0.5911260310726447 1loss/SRqqt1/detection_loss‚3.3106137545485246 /loss/FBt4/detection_loss‚1.7046134683557532 %loss/total_loss‚77.90356223218545 2loss/SRqqt1/assignment_loss‚2.5127171215258146 epoch‚1<ÆI^÷Ø 1loss/FBt1/assignment_loss‚0.39251982724848833 /loss/FRt3/detection_loss‚2.6086863511921585 trainer/global_step‚1649 0loss/FBt2/assignment_loss‚0.6412692666053772 1loss/SRqqt4/detection_loss‚1.0521723219881887 /loss/FRt3/assignment_loss‚5.987603922148009 0loss/FRt4/assignment_loss‚3.1496125058881166 1loss/FBt4/assignment_loss‚0.24224318367524492 /loss/FRt1/detection_loss‚3.0502207279205322 /loss/FRt4/detection_loss‚1.7701827770677105 _step‚134 /loss/FRt2/detection_loss‚2.2677211011094704 1loss/SRqqt3/detection_loss‚2.6858985280168466 .loss/FBt1/detection_loss‚2.277084704224225 _runtime‚ 1796.07975204 .loss/FRt2/assignment_loss‚4.75623522911753 2loss/SRqqt4/assignment_loss‚2.5738959053288335 0loss/FBt3/assignment_loss‚0.5911260310726447 1loss/SRqqt1/detection_loss‚3.3106137545485246 /loss/FBt4/detection_loss‚1.7046134683557532 %loss/total_loss‚77.90356223218545 2loss/SRqqt1/assignment_loss‚2.5127171215258146 epoch‚1 2loss/SRqqt3/assignment_loss‚2.5793152348748567 1loss/SRqqt2/detection_loss‚1.5670865545670192 .loss/FBt2/detection_loss‚20.00162625312805 .loss/FBt3/detection_loss‚3.352744793891907 % _timestamp‚1.7883967325199645e+09 0loss/FRt1/assignment_loss‚6.2488607035742865 1loss/SRqqt2/assignment_loss‚2.579511965115865†XÂúuø‚(:… À‰ãÔÙŒâÓ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åhmEù‚(:º À‰ãÔÚ¼Ö disk.drbd1777.out‚0 network.sent‚ 2.869187e+06 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2850.34765625) disk./.usagePercent‚72.56520604059813& disk.nvme0n1.out‚26306.81591796875( proc.memory.availableMB‚ 219.01953125 cpu‚25.224194158736136 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 1942.609375 network.recv‚662590 disk.rbd1.out‚ 0.919921875( disk.nvme0c0n1.out‚26306.81591796875 disk.nvme0n1.in‚ 334.48828125* proc.memory.percent‚17.397141456604004 disk.rbd2.out‚74.08349609375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 334.48828125 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.drbd1777.in‚0% disk./.usageGB‚1081.2740097045898 disk.rbd0.in‚0# disk.dm-0.out‚26306.81591796875 disk.sda.out‚ 1942.609375$ memory_percent‚98.66321086883545 disk.dm-1.out‚ 1942.609375 disk.dm-0.in‚ 334.48828125 disk.sda1.in‚0We0›ú‚(: ωãÔʆÊÓ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚96.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.261799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î&6»Eû‚(:º ωãÔÖ­ŸÖ% disk./.usageGB‚1081.2740898132324 disk.sda1.out‚0 disk.rbd2.out‚74.16162109375$ memory_percent‚98.66323471069336 cpu‚25.22278167331628 proc.cpu.threads‚17 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.397141456604004 disk.sda.out‚ 1962.95703125 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 1962.95703125' disk.nvme0c0n1.out‚26307.4189453125 disk.nvme0n1.in‚ 334.48828125% disk.nvme0n1.out‚26307.4189453125 disk.rbd3.in‚0 disk.rbd1.out‚ 0.919921875 disk.rbd0.in‚0 disk.dm-0.in‚ 334.48828125 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 219.015625 network.recv‚668346 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 1962.95703125" disk.dm-0.out‚26307.4189453125# proc.memory.rssMB‚ 2850.34765625) disk./.usagePercent‚72.56521141675634 network.sent‚ 2.976496e+06 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 334.48828125 disk.rbd0.out‚0ú6¥ü‚(:… Þ‰ãÔŽº‰Ó gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…çéJý‚(:¿ Þ‰ãÔ…³ÄÕ disk.nvme0n1.in‚ 334.48828125& disk.nvme0n1.out‚26307.78076171875 disk.rbd1.in‚0( disk.nvme0c0n1.out‚26307.78076171875 disk.dm-1.in‚ 0.07421875 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 0.98291015625 disk.rbd3.in‚0' proc.memory.availableMB‚ 219.0078125 disk.sda1.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1081.2741928100586 disk.dm-0.in‚ 334.48828125$ memory_percent‚98.66328239440918 network.sent‚ 2.98269e+06# disk.dm-0.out‚26307.78076171875 disk.sda.out‚ 1963.64453125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚75.67333984375 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875) disk./.usagePercent‚72.56521832895977 disk.sda3.out‚ 1963.64453125* proc.memory.percent‚17.397165298461914 cpu‚25.221540250754465 disk.drbd1777.out‚0 disk.rbd0.in‚0 network.recv‚672744" disk.nvme0c0n1.in‚ 334.48828125 disk.dm-1.out‚ 1963.64453125 disk.sda.in‚ 0.07421875" proc.memory.rssMB‚ 2850.3515625 disk.sda2.in‚0â¸ZÌþ‚(:… í‰ãÔ¢ÙÔ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.069' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6138 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0irë¨Eÿ‚(:º í‰ãÔÜÁ×# disk.dm-0.out‚26308.66748046875 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0( proc.memory.availableMB‚ 219.25390625 disk.rbd3.out‚0% disk./.usageGB‚1081.2742500305176 disk.sda3.out‚ 1981.171875" disk.nvme0c0n1.in‚ 334.48828125) disk./.usagePercent‚72.56522216907278 disk.rbd0.in‚0" proc.memory.rssMB‚ 2850.3515625 proc.cpu.threads‚17& disk.nvme0n1.out‚26308.66748046875 disk.dm-0.in‚ 334.48828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0$ memory_percent‚98.66178035736084 network.recv‚676797 disk.nvme0n1.in‚ 334.48828125 disk.rbd2.out‚75.81396484375 disk.rbd3.in‚0* proc.memory.percent‚17.397165298461914 cpu‚25.22012417311144 disk.rbd0.out‚0 disk.sda.out‚ 1981.171875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 0.98291015625 disk.dm-1.out‚ 1981.171875 network.sent‚ 2.990322e+06 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚26308.66748046875 disk.dm-1.in‚ 0.07421875w†@K2€- _wandbruntime‚1846  _runtime‚1846ñŸØÉà ' lr-AdamW/pg2‚0.00022190000156691915 trainer/global_step‚1699 % _timestamp‚1.7883967828527193e+09 _runtime‚1846.412579313 _step‚135 ' lr-AdamW/pg1‚0.00022190000156691915hrøÎ‚È ' lr-AdamW/pg1‚0.00022190000156691915 ' lr-AdamW/pg2‚0.00022190000156691915 trainer/global_step‚1699 % _timestamp‚1.7883967828527193e+09 _runtime‚1846.412579313 _step‚135‡ñÙ2ƒ- _wandbruntime‚1846  _runtime‚1846³2í„ÂO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31881921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30819917 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 _step‚136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31881923 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44983385 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64077394866067 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67945614960128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3096837248000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12121039 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3134408950272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3087269590559 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64849011296768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3096837248000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18699610 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44957205 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75525379690496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64849029299712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31881921 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67945866547712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67945866547712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64077148856863 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44983385 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12120309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60989879266304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67945614960128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293671936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18699608 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78659494968832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3096603663360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31881923 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 trainer/global_step‚1699CÁìÝÍ'…ÇO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64849029299712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31881921 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67945866547712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67945866547712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64077148856863 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44983385 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12120309 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60989879266304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67945614960128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293671936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18699608 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78659494968832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3096603663360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31881923 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 trainer/global_step‚1699 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31881921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30819917 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 _step‚136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31881923 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44983385 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64077394866067 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67945614960128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3096837248000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12121039 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3134408950272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3087269590559 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64849011296768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3096837248000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18699610 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44957205 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75525379690496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ˆOcÌ2†- _wandbruntime‚1847  _runtime‚1847ÂÔÑ&‡ËM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67984817999360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚137 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76883380 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18710263 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76909588 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30837341 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267621376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64886171589632 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67984817999360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 _runtime‚ 1847.39191141 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61024805959680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76909588 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67984566398464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45009486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3098412811776 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64114125293091 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7883967838312385e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64886153586688 trainer/global_step‚1699 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76883380 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3089302294051 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚137 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76883380 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18710263 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76909588 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30837341 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267621376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.larx¨­¤ge_pool.allocated‚64886171589632 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67984817999360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 _runtime‚ 1847.39191141 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61024805959680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76909588 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67984566398464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45009486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3098412811776 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64114125293091 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7883967838312385e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64886153586688 trainer/global_step‚1699 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76883380 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3089302294051 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12465421 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31698405 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚726 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77694301235200 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265537536 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79081360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62736230974464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32790873 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7883968348013752e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46290487 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80920156604416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46290487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3177698823611 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79055178 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32790871 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65913683788871 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32790871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚62736213935104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46264307 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69893452878336 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3177469853767 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69893452878336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32790873 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12466145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77694325175296 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289477632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 trainer/global_step‚1749 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69893201290752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66706143795712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚724 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3187309082624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3225565891584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19232258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2B¯¦mÏ"—ÆO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289477632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 trainer/global_step‚1749 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69893201290752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66706143795712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚724 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3187309082624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3225565891584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19232258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66706125792768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66706143795712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3187075497984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19232260 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚69893201290752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31697679 _runtime‚ 1898.36141491 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66706125792768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3225831429120 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65913929798075 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12465421 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31698405 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚726 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77694301235200 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265537536 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79081360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62736230974464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32790873 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7883968348013752e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46290487 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80920156604416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46290487 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3177698823611 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79055178 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32790871 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65913683788871 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32790871 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚62736213935104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46264307 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.allD ºˆý.allocated‚69893452878336 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3177469853767 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69893452878336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32790873 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12466145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77694325175296ŒºªGƒ2˜- _wandbruntime‚1899  _runtime‚1899Ç ´Ò&™ÌM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77737680144896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46316588 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69932152729088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚141 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32809050 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31714986 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66743286085632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80965084464128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12472782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12472075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 276881408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3227657260544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31715695 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65950660225099 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3188884646400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46290382 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32809052 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46316588 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 % _timestamp‚1.7883968357594578e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3188884646400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32809052 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3227404319232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62771157667840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚69932404329984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66743286085632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66743268082688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79125640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77737704084992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62771140628480 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79125640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46290382 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65950414215687 ðÑí×&šÑM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46316588 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69932152729088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚141 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32809050 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31714986 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66743286085632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80965084464128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12472782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12472075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 276881408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3227657260544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31715695 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65950660225099 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3188884646400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46290382 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32809052 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46316588 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 % _timestamp‚1.7883968357594578e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3188884646400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32809052 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3227404319232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62771157667840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚69932404329984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66743286085632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66743268082688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79125640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77737704084992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62771140628480 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79125640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46290382 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65950414215687 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77737680144896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206ÎÝÞÉ2›- _wandbruntime‚1899  _runtime‚1899ׯÐXœÒ _step‚142 0loss/FBt1/assignment_loss‚0.5657549175722166 0loss/FBt2/assignment_loss‚0.8063033390014122 0loss/FBt3/assignment_loss‚0.4922052267743169 1loss/FBt4/assignment_loss‚0.23119184137476623 /loss/FRt3/detection_loss‚2.6760969081440487 epoch‚1 1loss/SRqqt2/assignment_loss‚2.566210087607889 1loss/SRqqt4/detection_loss‚1.2362420643758343 /loss/FBt4/detection_loss‚1.7034003226594492 trainer/global_step‚1749 /loss/FRt2/assignment_loss‚4.311677201738897 %loss/total_loss‚63.42896675073651 $ _timestamp‚1.788396835761585e+09 0loss/SRqqt1/detection_loss‚3.229552034288645 /loss/FBt1/detection_loss‚1.4720656269184267 /loss/FRt1/detection_loss‚2.7959874130430675 /loss/FRt1/assignment_loss‚6.365907805306571 1loss/SRqqt2/detection_loss‚1.3550723522901535 0loss/SRqqt3/detection_loss‚3.481903038918972 /loss/FRt3/assignment_loss‚6.182261312330091 0loss/FRt4/assignment_loss‚3.3306972655383023 /loss/FBt3/detection_loss‚2.8056651279330254 2loss/SRqqt1/assignment_loss‚2.4564781188964844 2loss/SRqqt4/assignment_loss‚2.7762719263513405 /loss/FRt4/detection_loss‚2.0323281808332965 2loss/SRqqt3/assignment_loss‚2.9752491265535355 /loss/FRt2/detection_loss‚2.3183650813012755 .loss/FBt2/detection_loss‚5.262080430984497 _runtime‚1899.321313598Í9æ]× %loss/total_loss‚63.42896675073651 $ _timestamp‚1.788396835761585e+09 0loss/SRqqt1/detection_loss‚3.229552034288645 /loss/FBt1/detection_loss‚1.4720656269184267 /loss/FRt1/detection_loss‚2.7959874130430675 /loss/FRt1/assignment_loss‚6.365907805306571 1loss/SRqqt2/detection_loss‚1.3550723522901535 0loss/SRqqt3/detection_loss‚3.481903038918972 /loss/FRt3/assignment_loss‚6.182261312330091 0loss/FRt4/assignment_loss‚3.3306972655383023 /loss/FBt3/detection_loss‚2.8056651279330254 2loss/SRqqt1/assignment_loss‚2.4564781188964844 2loss/SRqqt4/assignment_loss‚2.7762719263513405 /loss/FRt4/detection_loss‚2.0323281808332965 2loss/SRqqt3/assignment_loss‚2.9752491265535355 /loss/FRt2/detection_loss‚2.3183650813012755 .loss/FBt2/detection_loss‚5.262080430984497 _runtime‚1899.321313598 _step‚142 0loss/FBt1/assignment_loss‚0.5657549175722166 0loss/FBt2/assignment_loss‚0.8063033390014122 0loss/FBt3/assignment_loss‚0.4922052267743169 1loss/FBt4/assignment_loss‚0.23119184137476623 /loss/FRt3/detection_loss‚2.6760969081440487 epoch‚1 1loss/SRqqt2/assignment_loss‚2.566210087607889 1loss/SRqqt4/detection_loss‚1.2362420643758343 /loss/FBt4/detection_loss‚1.7034003226594492 trainer/global_step‚1749 /loss/FRt2/assignment_loss‚4.311677201738897Ž#­DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68563258291712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83180605536256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_stzç\õkart/requested_bytes.small_pool.allocated‚ 3267899086819 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81271230 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71840787621376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33699823 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81297412 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291574784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67750464730083 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47597589 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33699821 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79863270660096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67750218720879 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3267670116975 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 trainer/global_step‚1799 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68563258291712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3277780917248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12808988 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 % _timestamp‚1.7883968859408062e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68563240288768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19764908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3277547332608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚727 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47597589 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71840787621376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3277780917248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3317334876160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64482565643264 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚71841039208960 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚71841039208960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚725 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19764910 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267634688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68563240288768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83180313961472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79863246720000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360Q–á1Í'©ÇO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚71841039208960 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚71841039208960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚725 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19764910 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267634688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68563240288768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83180313961472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79863246720000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33699823 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12809713 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 _step‚144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33699821 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47571409 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68563258291712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83180605536256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3267899086819 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81271230 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71840787621376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33699823 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81297412 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291574784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67750464730083 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47597589 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33699821 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79863270660096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67750218720879 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3267670116975 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 trainer/global_step‚1799 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68563258291712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3277780917248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12808988 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 % _timestamp‚1.7883968859408062e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68563240288768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19764908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3277547332608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚727 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47597589 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71840787621376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3277780917248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3317334876160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64482565643264;ˆ ˆª‚(:… ÖŠãÔ¬ÐàÓ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚99.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8026 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò&.B«‚(:· ÖŠãÔ¬Œ­Ö% disk./.usageGB‚1081.2823753356934 disk.sda.out‚ 2023.1328125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 218.5703125 proc.cpu.threads‚17 network.recv‚709987) disk./.usagePercent‚72.56576746512071 disk.dm-1.out‚ 2023.1328125 disk.rbd2.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.397165298461914 disk.rbd2.out‚ 77.716796875 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 disk.sda1.in‚0" proc.memory.rssMB‚ 2850.3515625' disk.nvme0c0n1.out‚26320.7783203125% disk.nvme0n1.out‚26320.7783203125 disk.sda2.in‚0 disk.sda3.out‚ 2023.1328125 disk.dm-0.in‚ 335.26171875 disk.dm-1.in‚ 0.07421875 disk.rbd3.out‚0$ memory_percent‚98.66595268249512 network.sent‚ 3.233395e+06" disk.dm-0.out‚26320.7783203125 disk.sda1.out‚0 disk.rbd1.out‚ 0.98291015625 cpu‚25.21135741559374" disk.nvme0c0n1.in‚ 335.26171875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 335.26171875•g1­2¬- _wandbruntime‚1950  _runtime‚1950l¡„fÒ&­ÌM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68600382578688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81341692 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚71879990660608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83225812284416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33718000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19775563 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚722 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚71879739059712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67787195157107 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32591124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12815563 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83225520722944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33718002 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79906625629696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3319162714624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64517492336640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68600400581632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64517475297280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81315484 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3279356481024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12816285 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267621376 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚71879990660608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3279356481024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚1799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47597484 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚724 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 _step‚145 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33718002 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81341692 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3279590078976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33718000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68600400581632 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47623690 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841lÔ2·Á®ÑM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32591848 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64517492336640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68600400581632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64517475297280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81315484 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3279356481024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12816285 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267621376 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚71879990660608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3279356481024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚1799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47597484 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚724 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 _step‚145 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33718002 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81341692 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3279590078976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33718000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 XIDeviceStatsMonitor.on_train_batch_end/rld“ñeserved_bytes.large_pool.allocated‚ 6505365504 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68600400581632 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47623690 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68600382578688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81341692 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚71879990660608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83225812284416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33718000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19775563 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚722 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚71879739059712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67787195157107 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32591124 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12815563 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83225520722944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33718002 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79906625629696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3319162714624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 Fµ‚(:³ ôŠãÔ–…ÏÖ disk.dm-0.in‚ 335.26171875 disk.sda.in‚ 0.07421875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 2077.23828125 disk.drbd1777.out‚0 network.recv‚721838 disk.nvme0n1.in‚ 335.26171875 disk.dm-0.out‚ 26421.4609375 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 26421.4609375 disk.dm-1.out‚ 2077.23828125 disk.rbd0.in‚0) disk./.usagePercent‚72.57416784034174 disk.sda2.in‚0 disk.dm-1.in‚ 0.07421875' proc.memory.availableMB‚ 217.5703125 cpu‚25.208937805957532 network.sent‚ 3.427124e+06" disk.nvme0c0n1.in‚ 335.26171875 disk.sda.out‚ 2077.23828125 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 77.8271484375 disk.rbd0.out‚0$ memory_percent‚98.67205619812012 disk.sda1.out‚0% disk./.usageGB‚1081.4075469970703" disk.nvme0n1.out‚ 26421.4609375 disk.sda1.in‚0 disk.sda3.in‚ 0.07421875 disk.sda2.out‚0 disk.rbd1.out‚ 0.98291015625" proc.memory.rssMB‚ 2850.3515625* proc.memory.percent‚17.397165298461914 proc.cpu.threads‚17UT‰É›¶‚(: ƒ‹ãÔÀÓ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚76.485' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.296999999999999 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y‘‘!E·‚(:º ƒ‹ãÔÔ¬ÜÖ network.sent‚ 3.432397e+06 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 2095.2734375# disk.dm-0.out‚26421.89208984375% disk./.usageGB‚1081.4076080322266 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 335.26171875* proc.memory.percent‚17.397165298461914 cpu‚25.2078794588927 network.recv‚725401 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 335.26171875 disk.sda1.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 335.26171875 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚72.57417193646228 disk.sda.out‚ 2095.2734375& disk.nvme0n1.out‚26421.89208984375 disk.rbd2.out‚ 79.4794921875 disk.sda.in‚ 0.07421875 disk.sda1.in‚0 disk.sda3.out‚ 2095.2734375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.98291015625( disk.nvme0c0n1.out‚26421.89208984375# memory_percent‚98.6706018447876 disk.sda3.in‚ 0.07421875( proc.memory.availableMB‚ 217.80859375" proc.memory.rssMB‚ 2850.3515625;†t2¸- _wandbruntime‚2002  _runtime‚2002,x®¸É¹Ã ' lr-AdamW/pg1‚0.00022111577375653958 ' lr-AdamW/pg2‚0.00022111577375653958 trainer/global_step‚1849 % _timestamp‚1.7883969385119386e+09 _runtime‚2002.071794595 _step‚147”º(RÎºÈ ' lr-AdamW/pg1‚0.00022111577375653958 ' lr-AdamW/pg2‚0.00022111577375653958 trainer/global_step‚1849 % _timestamp‚1.7883969385119386e+09 _runtime‚2002.071794595 _step‚147“Oñš>2»- _wandbruntime‚2002  _runtime‚2002b¤GNÈ'¼ÂO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48878511 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48878511 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70420354784768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 % _timestamp‚1.7883969385127153e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48904691 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33450518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20297560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73788625539584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3368019167232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85440959366144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3408743221248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358099350027 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3368019167232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82032216144896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34608771 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13153659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 287380480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34608773 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73788373952000 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73788625539584 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73788373952000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48904691 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66228900312064 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70420372787712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83513464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20297558 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0¯éØi$½ÇO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34608771 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34608771 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13153659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 287380480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34608773 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73788373952000 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73788625539584 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73788373952000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48904691 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66228900312064 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70420372787712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83513464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20297558 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48878511 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48878511 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70420354784768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 % _timestamp‚1.7883969385127153e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48904691 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33450518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20297560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73788625539584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3368019167232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85440959366144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3408743221248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358099350027 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3368019167232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82032216144896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3369828315648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 % _timestamp‚1.7883969394836547e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34626952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3369828315648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 _runtime‚2003.044003016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66263809966080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66263827005440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48930792 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73827576991232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70457515077632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34626952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69623730089115 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚149 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73827576991232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48930792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82075571114496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34626950 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370061913600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83531536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3410305644544 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20308213 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚726 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3359674113623 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83531536 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34626950 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3359903083675 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33468299 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85486166223360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13160814 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69623484079703 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370061913600 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85485876759040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48904586 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48904586 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83557744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70457515077632 trainer/global_step‚1849 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83557744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33468299 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85486166223360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13160814 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69623484079703 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370061913600 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85485876759040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48904586 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48904586 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83557744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70457515077632 trainer/global_step‚1849 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83557744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3369828315648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 % _timestamp‚1.7883969394836547e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34626952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3369828315648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 _runtime‚2003.044003016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66263809966080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66263827005440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48930792 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73827576991232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70457515077632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34626952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69623730089115 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚149 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73827576991232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48930792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82075571114496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34626950 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370061913600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83531536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3410305644544 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20308213 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚726 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3359674113623 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83531536 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34626950 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3359903083675 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72277469280768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3500163638272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3499910683648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50211793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3458491001856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 252954624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75736211870208 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚75735960282624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75735960282624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72277487283712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87701325267968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34328640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50185613 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72277469280768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3500163638272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3499910683648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50211793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3458491001856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 252954624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75736211870208 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚75735960282624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75735960282624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72277487283712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87701325267968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34328640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50185613 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35535902 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7883969888278968e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚75775163321856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2052.388394914 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75775163321856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3460300150272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85773796 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85773796 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50237894 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87746542149632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50237894 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68010144634880 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72314611570688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3460533748224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85747588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34345628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85747588 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34346343 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68010161674240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3460533748224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20840863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13505480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72314629573632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3501729794560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚75774911720960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72314629573632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3502001610240 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 _step‚153 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20840861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3449874376831 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50211688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35535900 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35535900 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13504767 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35535902 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71460019011711 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50211688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295755776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3450103346883 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504Ì uÞPÒÑM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20840861 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3449874376831 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50211688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35535900 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35535900 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13504767 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35535902 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71460019011711 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50211688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295755776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3450103346883 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72314611570688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84244540539392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35535902 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7883969888278968e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚75775163321856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2052.388394914 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75775163321856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3460300150272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85773796 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85773796 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50237894 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87746542149632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50237894 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68010144634880 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72314611570688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3460533748224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85747588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34345628 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85747588 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34346343 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68010161674240 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3460533748224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20840863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13505480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72314629573632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3501729794560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚75774911720960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72314629573632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3502001610240 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 _step‚153™½*KŠ2Ó- _wandbruntime‚2052  _runtime‚2052Ô~µÔTÔÎ 0loss/FBt4/assignment_loss‚0.4196508872400128 1loss/SRqqt2/detection_loss‚1.6492574362782761 0loss/FRt4/assignment_loss‚3.0994816701258383 2loss/SRqqt4/assignment_loss‚2.4174070244139814 _step‚154 1loss/SRqqt1/assignment_loss‚2.873702755996159 1loss/SRqqt3/detection_loss‚2.8767102101574773 .loss/FBt2/detection_loss‚7.810866594314575 .loss/FBt3/detection_loss‚2.803417823144368 % _timestamp‚1.7883969888296976e+09 1loss/SRqqt2/assignment_loss‚2.490955944173038 1loss/SRqqt3/assignment_loss‚2.658733896587206 epoch‚1 _runtime‚2052.389404406 /loss/FRt2/assignment_loss‚4.639347290620208 /loss/FRt3/assignment_loss‚6.260887427763506 .loss/FRt2/detection_loss‚2.164788686670363 /loss/FRt4/detection_loss‚1.8652002619158836 0loss/FBt2/assignment_loss‚2.1821651458740234 .loss/FRt1/detection_loss‚2.540819482803345 /loss/FBt1/detection_loss‚1.3246345456820707 trainer/global_step‚1899 1loss/FBt1/assignment_loss‚0.48198906268474645 /loss/FRt3/detection_loss‚2.3760147750038993 1loss/SRqqt1/detection_loss‚3.6777954953057423 1loss/SRqqt4/detection_loss‚1.1535111452512286 .loss/FBt4/detection_loss‚1.417034266855229 %loss/total_loss‚65.95075533991742 .loss/FRt1/assignment_loss‚6.24795970916748 0loss/FBt3/assignment_loss‚0.5184238018887625NŽ—YÕÓ %loss/total_loss‚65.95075533991742 .loss/FRt1/assignment_loss‚6.24795970916748 0loss/FBt3/assignment_loss‚0.5184238018887625 0loss/FBt4/assignment_loss‚0.4196508872400128 1loss/SRqqt2/detection_loss‚1.6492574362782761 0loss/FRt4/assignment_loss‚3.0994816701258383 2loss/SRqqt4/assignment_loss‚2.4174070244139814 _step‚154 1loss/SRqqt1/assignment_loss‚2.873702755996159 1loss/SRqqt3/detection_loss‚2.8767102101574773 .loss/FBt2/detection_loss‚7.810866594314575 .loss/FBt3/detection_loss‚2.803417823144368 % _timestamp‚1.7883969888296976e+09 1loss/SRqqt2/assignment_loss‚2.490955944173038 1loss/SRqqt3/assignment_loss‚2.658733896587206 epoch‚1 _runtime‚2052.389404406 /loss/FRt2/assignment_loss‚4.639347290620208 /loss/FRt3/assignment_loss‚6.260887427763506 .loss/FRt2/detection_loss‚2.164788686670363 /loss/FRt4/detection_loss‚1.8652002619158836 0loss/FBt2/assignment_loss‚2.1821651458740234 .loss/FRt1/detection_loss‚2.540819482803345 /loss/FBt1/detection_loss‚1.3246345456820707 trainer/global_step‚1899 1loss/FBt1/assignment_loss‚0.48198906268474645 /loss/FRt3/detection_loss‚2.3760147750038993 1loss/SRqqt1/detection_loss‚3.6777954953057423 1loss/SRqqt4/detection_loss‚1.1535111452512286 .loss/FBt4/detection_loss‚1.417034266855229šÅ%ŽÖ‚(:ƒ ¿‹ãÔßÕØÔ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚83.94' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.788 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ›4Iׂ(:¾ ¿‹ãÔüùÆÖ# disk.dm-0.out‚30733.23681640625 network.sent‚ 3.557825e+06% disk./.usageGB‚1079.6678428649902 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 350.23046875 disk.dm-0.in‚ 350.23046875& disk.nvme0n1.out‚30748.36181640625# proc.memory.rssMB‚ 2850.35546875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 network.recv‚744444) disk./.usagePercent‚72.45741483632885 disk.sda3.out‚ 2186.4609375 disk.dm-1.in‚ 0.07421875' proc.memory.availableMB‚ 217.6015625 disk.rbd2.out‚79.74169921875" disk.nvme0c0n1.in‚ 350.23046875 disk.dm-1.out‚ 2186.4609375 disk.rbd0.in‚0* proc.memory.percent‚17.397189140319824 disk.sda1.out‚0 disk.drbd1777.in‚0( disk.nvme0c0n1.out‚30747.98681640625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 0.98291015625 disk.sda.in‚ 0.07421875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.out‚ 2186.4609375$ memory_percent‚98.67186546325684 cpu‚25.203452899332245iëóœØ‚(:‘ ΋ãÔÏ£ŽÓ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.913' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.982599999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0” >0Ù‚(:¥ ΋ãÔÎÀñÖ disk.rbd2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚394.6875 disk.rbd0.out‚0 disk.nvme0n1.in‚394.6875$ disk./.usageGB‚1081.742877960205 disk.drbd1777.in‚0 disk.rbd2.out‚79.74169921875 disk.sda3.in‚ 0.07421875$ memory_percent‚98.67196083068848 disk.sda.out‚2187.375 disk.sda2.in‚0 disk.sda3.out‚2187.375# proc.memory.rssMB‚ 2850.35546875* proc.memory.percent‚17.397189140319824) disk./.usagePercent‚72.59667218263914 disk.dm-0.in‚394.6875 disk.sda1.out‚0 disk.rbd1.out‚ 0.98291015625 disk.dm-1.out‚2187.375 network.recv‚750991# disk.dm-0.out‚34148.90869140625 disk.sda.in‚ 0.07421875 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚34148.90869140625 disk.rbd0.in‚0' proc.memory.availableMB‚ 217.5859375 network.sent‚ 3.666013e+06 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875 cpu‚25.202466237426943& disk.nvme0n1.out‚34148.90869140625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.out‚0Ùð÷3‘Ú‚(:† Ý‹ãÔäÁßÒ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0396 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õºÈBFÛ‚(:» Ý‹ãÔ®¯¾Ö disk.sda.in‚ 0.07421875) disk./.usagePercent‚72.60323519178692 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 2187.8515625( proc.memory.availableMB‚ 217.57421875 cpu‚25.201393195764144 disk.rbd2.out‚81.45654296875 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 0.98291015625 disk.dm-1.out‚ 2187.8515625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 482.4765625* proc.memory.percent‚17.397189140319824 network.sent‚ 3.670513e+06 disk.rbd2.in‚0 disk.dm-1.in‚ 0.07421875# memory_percent‚98.6720323562622 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 2187.8515625! disk.nvme0c0n1.in‚ 482.4765625& disk.nvme0n1.out‚35505.29443359375 proc.cpu.threads‚17# disk.dm-0.out‚35505.29443359375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚35505.29443359375# proc.memory.rssMB‚ 2850.35546875 disk.dm-0.in‚ 482.4765625 network.recv‚754160% disk./.usageGB‚1081.8406715393066 disk.rbd3.out‚0ªkËsÜ‚(:… ì‹ãÔßËåÓ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚83.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.6232 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðdBÝ‚(:· ì‹ãÔ½×œÖ disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 502.34375& disk.nvme0n1.out‚35768.73974609375* proc.memory.percent‚17.397189140319824 network.sent‚ 3.676844e+06 network.recv‚758966 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚35768.73974609375 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 0.98291015625 disk.sda.out‚ 2188.50390625 disk.dm-0.in‚ 502.34375 disk.sda.in‚ 0.07421875) disk./.usagePercent‚72.60293131084377% disk./.usageGB‚1081.8361434936523 disk.rbd3.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2188.50390625 disk.rbd2.out‚81.45654296875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚98.6720323562622( proc.memory.availableMB‚ 217.57421875# proc.memory.rssMB‚ 2850.35546875 proc.cpu.threads‚17 disk.sda3.out‚ 2188.50390625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 502.34375 disk.drbd1777.in‚0 cpu‚25.20033165649948 disk.rbd0.in‚0 disk.sda3.in‚ 0.07421875( disk.nvme0c0n1.out‚35768.73974609375Ø¿$2Þ- _wandbruntime‚2101  _runtime‚21012X[Éßà ' lr-AdamW/pg2‚0.00022047865290461403 trainer/global_step‚1949 % _timestamp‚1.7883970381785479e+09 _runtime‚2101.738368032 _step‚155 ' lr-AdamW/pg1‚0.00022047865290461403+’ÎàÈ ' lr-AdamW/pg1‚0.00022047865290461403 ' lr-AdamW/pg2‚0.00022047865290461403 trainer/global_step‚1949 % _timestamp‚1.7883970381785479e+09 _runtime‚2101.738368032 _step‚155›ûq%í2á- _wandbruntime‚2101  _runtime‚2101»éÈ'âÂO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3548962836480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 261343232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73260069526107 % _timestamp‚1.7883970381794271e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3549196421120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74134601779712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3548962836480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 _step‚156 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35208023 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77683546613248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74134601779712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 _runtime‚2101.739558746 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51518895 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89961626294272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36426671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51492715 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89961341010944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51492715 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13845165 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3591257836544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73260069526107 % _timestamp‚1.7883970381794271e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3549196421120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74134601779712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3548962836480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 _step‚156 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35208023 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77683546613248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74134601779712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 _runtime‚2101.739558746 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51518895 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89961626294272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36426671 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51492715 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89961341010944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51492715 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13845165 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3591257836544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35225508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3593084996608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51544996 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74171726066688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86413462084096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35226212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13851997 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90006832350720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51544996 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51518790 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36444850 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74171744069632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3551005582848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87963640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74171744069632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚1949 _step‚157 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87963640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36444850 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21373513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21373511 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 _runtime‚2102.769079133 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36444852 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74171726066688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90006547080704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.788397039208596e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3550771984896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87989848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3551005582848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13852699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36444852 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253y‹þ”Ö&æÐM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3551005582848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13852699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36444852 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚77722749652480 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35225508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3593084996608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51544996 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74171726066688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86413462084096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35226212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13851997 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90006832350720 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51544996 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51518790 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36444850 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚74171744069632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3551005582848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87963640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74171744069632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚1949 _step‚157 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87963640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36444850 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21373513 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21373511 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 _runtime‚2102.769079133 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36444852 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74171726066688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90006547080704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.788397039208596e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3550771984896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87989848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456cƒ·N2ç- _wandbruntime‚2102  _runtime‚2102ª_Œ TèÎ /loss/FRt4/assignment_loss‚2.775303892673008 /loss/FRt4/detection_loss‚1.9659054142142098 .loss/FBt3/detection_loss‚2.523516630425173 epoch‚1 _runtime‚2102.770400563 2loss/SRqqt2/assignment_loss‚2.5682762265205383 2loss/SRqqt4/assignment_loss‚2.6127872316391914 0loss/FBt4/assignment_loss‚0.4742156737636833 /loss/FRt1/detection_loss‚2.9953844856332847 .loss/FBt2/detection_loss‚4.215421756108602 _step‚158 0loss/SRqqt1/detection_loss‚3.211222488965307 $loss/total_loss‚62.1076552437044 /loss/FRt2/assignment_loss‚4.702230650803139 /loss/FRt3/assignment_loss‚6.261631806691487 /loss/FRt3/detection_loss‚2.4493438319436143 1loss/SRqqt2/detection_loss‚1.6245501819448394 /loss/FBt1/detection_loss‚1.4005561682669592 0loss/SRqqt4/detection_loss‚1.070036597508978 trainer/global_step‚1949 /loss/FRt1/assignment_loss‚6.040771254786739 2loss/SRqqt3/assignment_loss‚3.1502337634563444 /loss/FRt2/detection_loss‚2.1741990703960945 -loss/FBt4/detection_loss‚1.68764701962471 % _timestamp‚1.7883970392106066e+09 1loss/SRqqt1/assignment_loss‚2.880011405263628 1loss/FBt1/assignment_loss‚0.46896338492554906 0loss/FBt2/assignment_loss‚0.8887951970100403 0loss/FBt3/assignment_loss‚0.6520587854518667 1loss/SRqqt3/detection_loss‚3.3145923256874084%ýˆyYéÓ 2loss/SRqqt2/assignment_loss‚2.5682762265205383 2loss/SRqqt4/assignment_loss‚2.6127872316391914 0loss/FBt4/assignment_loss‚0.4742156737636833 /loss/FRt1/detection_loss‚2.9953844856332847 .loss/FBt2/detection_loss‚4.215421756108602 _step‚158 0loss/SRqqt1/detection_loss‚3.211222488965307 $loss/total_loss‚62.1076552437044 /loss/FRt2/assignment_loss‚4.702230650803139 /loss/FRt3/assignment_loss‚6.261631806691487 /loss/FRt3/detection_loss‚2.4493438319436143 1loss/SRqqt2/detection_loss‚1.6245501819448394 /loss/FBt1/detection_loss‚1.4005561682669592 0loss/SRqqt4/detection_loss‚1.070036597508978 trainer/global_step‚1949 /loss/FRt1/assignment_loss‚6.040771254786739 2loss/SRqqt3/assignment_loss‚3.1502337634563444 /loss/FRt2/detection_loss‚2.1741990703960945 -loss/FBt4/detection_loss‚1.68764701962471 % _timestamp‚1.7883970392106066e+09 1loss/SRqqt1/assignment_loss‚2.880011405263628 1loss/FBt1/assignment_loss‚0.46896338492554906 0loss/FBt2/assignment_loss‚0.8887951970100403 0loss/FBt3/assignment_loss‚0.6520587854518667 1loss/SRqqt3/detection_loss‚3.3145923256874084 /loss/FRt4/assignment_loss‚2.775303892673008 /loss/FRt4/detection_loss‚1.9659054142142098 .loss/FBt3/detection_loss‚2.523516630425173 epoch‚1 _runtime‚2102.770400563ž¹¸ê‚(:… û‹ãÔ²ÕÔ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Síx ?ë‚(:´ û‹ãÔ˜ßÂÖ network.recv‚762527 disk.dm-1.out‚ 2210.24609375# disk.dm-0.out‚35847.97412109375# proc.memory.rssMB‚ 2850.35546875 cpu‚25.199388060935405 disk.rbd2.in‚0 disk.rbd2.out‚ 81.54296875 disk.sda3.in‚ 0.07421875& disk.nvme0n1.out‚35847.97412109375 disk.drbd1777.out‚0' proc.memory.availableMB‚ 217.5390625 disk.sda3.out‚ 2210.24609375 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 502.34375 disk.drbd1777.in‚0# memory_percent‚98.6722469329834 disk.rbd1.in‚0 network.sent‚ 3.695134e+06) disk./.usagePercent‚72.59974708913289% disk./.usageGB‚1081.7886962890625 disk.nvme0c0n1.in‚ 502.34375( disk.nvme0c0n1.out‚35847.97412109375* proc.memory.percent‚17.397189140319824 proc.cpu.threads‚17 disk.rbd1.out‚ 0.98291015625 disk.nvme0n1.in‚ 502.34375 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2210.24609375~7Ïì‚(:‚ ŠŒãÔŸ‘èÓ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.82' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.964 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ÍA<í‚(:± ŠŒãÔä´˜Ö disk.dm-1.out‚ 2211.6484375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2850.35546875 network.sent‚ 3.790228e+06 disk.rbd3.out‚0 disk.dm-1.in‚ 0.07421875 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.in‚ 0.07421875) disk./.usagePercent‚72.59815933040554 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.98291015625$ memory_percent‚98.67231845855713* proc.memory.percent‚17.397189140319824 disk.rbd2.in‚0 disk.rbd2.out‚ 81.55859375 disk.nvme0n1.in‚ 502.34375 cpu‚25.19835741689412$ disk./.usageGB‚1081.765037536621 disk.sda3.out‚ 2211.6484375 disk.nvme0c0n1.in‚ 502.34375 disk.rbd0.out‚0 disk.sda.out‚ 2211.6484375( proc.memory.availableMB‚ 217.52734375 proc.cpu.threads‚17 disk.sda1.out‚0( disk.nvme0c0n1.out‚35952.51611328125& disk.nvme0n1.out‚35952.51611328125 disk.dm-0.in‚ 502.34375# disk.dm-0.out‚35952.51611328125 network.recv‚768810 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0¬K´.î‚(:… ™ŒãÔ“Ž—Ó gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚87.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5416 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G&#/;ï‚(:° ™ŒãÔÁê„Ö disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd1.out‚ 0.98291015625 disk.dm-1.out‚ 2212.2109375& proc.memory.availableMB‚ 217.765625 network.sent‚ 3.795744e+06 disk.nvme0n1.in‚ 502.34375 disk.dm-1.in‚ 0.07421875% disk./.usageGB‚1081.7651138305664& disk.nvme0n1.out‚35954.37158203125 disk.sda.out‚ 2212.2109375 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.397189140319824 network.recv‚773060) disk./.usagePercent‚72.59816445055623# disk.dm-0.out‚35954.37158203125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0$ memory_percent‚98.67086410522461 disk.rbd2.in‚0 disk.rbd2.out‚ 81.62109375 disk.sda3.out‚ 2212.2109375 disk.nvme0c0n1.in‚ 502.34375( disk.nvme0c0n1.out‚35954.37158203125# proc.memory.rssMB‚ 2850.35546875 cpu‚25.19733591132916 disk.dm-0.in‚ 502.34375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0µÞC2ð- _wandbruntime‚2155  _runtime‚2155?Ö[óÉñà ' lr-AdamW/pg1‚0.00022012599381144443 ' lr-AdamW/pg2‚0.00022012599381144443 trainer/global_step‚1999 % _timestamp‚1.7883970921205447e+09 _runtime‚2155.680432972 _step‚159Öªô¾ÎòÈ _step‚159 ' lr-AdamW/pg1‚0.00022012599381144443 ' lr-AdamW/pg2‚0.00022012599381144443 trainer/global_step‚1999 % _timestamp‚1.7883970921205447e+09 _runtime‚2155.680432972ŸÁ© 2ó- _wandbruntime‚2155  _runtime‚2155ka­H¬ôÂO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90135438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3628700139651 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 % _timestamp‚1.7883970921215181e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14189606 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3639434671104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88539052599296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3628471169807 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75991698272768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71467904318464 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3639668255744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88539028659200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75096358448911 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37335621 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52825997 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37335621 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14188912 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92222009353216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 _step‚160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36084420 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36085116 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79631132943872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75991716275712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21895508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 trainer/global_step‚1999 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92221715681280 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79631132943872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21895510 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2155.681638945 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75991716275712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3682687022080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354¸ųÍ'õÇO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3639668255744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88539028659200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75096358448911 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37335621 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52825997 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37335621 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14188912 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92222009353216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 _step‚160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36084420 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36085116 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79631132943872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75991716275712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21895508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 trainer/global_step‚1999 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92221715681280 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79631132943872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21895510 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2155.681638945 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75991716275712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3682687022080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90135438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3628700139651 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 % _timestamp‚1.7883970921215181e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14189606 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3639434671104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88539052599296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3628471169807 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75991698272768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71467904318464 -,„à2ö- _wandbruntime‚2156  _runtime‚2156›Í^Ò&÷ÌM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76028840562688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚161 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52852098 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37353800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90179692 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3630503873299 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88582407568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79670084382208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚75133334885139 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79670335983104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88582431508992 _runtime‚2156.702407481 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37353802 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3684519618560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90205900 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21906161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76028840562688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37353800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3684780948480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3630274903247 trainer/global_step‚1999 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76028858565632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3641243819520 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52852098 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52825892 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3641477417472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14195609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36102485 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37353802 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79670335983104 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92267212457472 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71502831011840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76028858565632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90179692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14196322 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75133088875727 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71502813972480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90205900 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 % _timestamp‚1.7883970931418304e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3641243819520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92266927187456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36101770 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21906163 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚713 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Œ÷U6ðøÑM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79670084382208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚75133334885139 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79670335983104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88582431508992 _runtime‚2156.702407481 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37353802 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3684519618560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90205900 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21906161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76028840562688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37353800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3684780948480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3630274903247 trainer/global_step‚1999 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76028858565632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3641243819520 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52852098 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52825892 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3641477417472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14195609 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36102485 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37353802 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79670335983104 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92267212457472 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71502831011840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76028858565632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90179692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14196322 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75133088875727 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71502813972480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90205900 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 % _timestamp‚1.7883970931418304e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3641243819520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 ‰³Êêç WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92266927187456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36101770 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21906163 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚713 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76028840562688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚161 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52852098 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37353800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90179692 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3630503873299 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88582407568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¡È€ [2ù- _wandbruntime‚2156  _runtime‚2156i®2!RúÌ /loss/FRt4/assignment_loss‚3.202360645228741 0loss/FBt2/assignment_loss‚0.3424583828309551 1loss/SRqqt4/detection_loss‚1.0431785774230957 %loss/total_loss‚59.64825536834516 /loss/FRt1/assignment_loss‚6.416466024186876 1loss/SRqqt3/assignment_loss‚2.687392075856527 0loss/FBt4/assignment_loss‚0.3059297379336118 1loss/SRqqt2/detection_loss‚1.5276462015094636 1loss/SRqqt4/assignment_loss‚2.657344387769699 0loss/FBt1/assignment_loss‚0.4836615865074453 /loss/FRt1/detection_loss‚3.2058818406528897 epoch‚1 -loss/FRt2/detection_loss‚2.00790775996266 1loss/SRqqt1/detection_loss‚3.0833873521713984 .loss/FBt2/detection_loss‚3.021725594997406 trainer/global_step‚1999 /loss/FRt2/assignment_loss‚4.305321734963042 /loss/FBt1/detection_loss‚1.4265244362371357 % _timestamp‚1.7883970931440299e+09 _runtime‚2156.703785228 _step‚162 0loss/FBt3/assignment_loss‚0.6539248457085285 .loss/FRt4/detection_loss‚2.035325615429411 .loss/FRt3/detection_loss‚2.607816405997083 1loss/SRqqt3/detection_loss‚2.7297768239621765 /loss/FBt3/detection_loss‚2.8131344318389893 /loss/FRt3/assignment_loss‚6.015129553305136 1loss/SRqqt1/assignment_loss‚2.998450614157177 2loss/SRqqt2/assignment_loss‚2.5392850426178946 .loss/FBt4/detection_loss‚1.538225697097825_ÓÆ·WûÑ _runtime‚2156.703785228 _step‚162 0loss/FBt3/assignment_loss‚0.6539248457085285 .loss/FRt4/detection_loss‚2.035325615429411 .loss/FRt3/detection_loss‚2.607816405997083 1loss/SRqqt3/detection_loss‚2.7297768239621765 /loss/FBt3/detection_loss‚2.8131344318389893 /loss/FRt3/assignment_loss‚6.015129553305136 1loss/SRqqt1/assignment_loss‚2.998450614157177 2loss/SRqqt2/assignment_loss‚2.5392850426178946 .loss/FBt4/detection_loss‚1.538225697097825 /loss/FRt4/assignment_loss‚3.202360645228741 0loss/FBt2/assignment_loss‚0.3424583828309551 1loss/SRqqt4/detection_loss‚1.0431785774230957 %loss/total_loss‚59.64825536834516 /loss/FRt1/assignment_loss‚6.416466024186876 1loss/SRqqt3/assignment_loss‚2.687392075856527 0loss/FBt4/assignment_loss‚0.3059297379336118 1loss/SRqqt2/detection_loss‚1.5276462015094636 1loss/SRqqt4/assignment_loss‚2.657344387769699 0loss/FBt1/assignment_loss‚0.4836615865074453 /loss/FRt1/detection_loss‚3.2058818406528897 epoch‚1 -loss/FRt2/detection_loss‚2.00790775996266 1loss/SRqqt1/detection_loss‚3.0833873521713984 .loss/FBt2/detection_loss‚3.021725594997406 trainer/global_step‚1999 /loss/FRt2/assignment_loss‚4.305321734963042 /loss/FBt1/detection_loss‚1.4265244362371357 % _timestamp‚1.7883970931440299e+09¢ãMñ¬›ü‚(: ¨ŒãÔ‚ÉÓ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚96.549' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.309800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[‰;¬6ý‚(:« ¨ŒãÔÏá‰Ö disk.sda3.out‚ 2213.1328125 disk.rbd1.out‚ 0.98291015625 disk.nvme0c0n1.in‚ 502.34375( disk.nvme0c0n1.out‚35954.73291015625 disk.dm-1.out‚ 2213.1328125 disk.dm-0.in‚ 502.34375$ memory_percent‚98.67191314697266! proc.memory.rssMB‚ 2850.359375$ disk./.usageGB‚1082.016128540039 disk.rbd2.in‚0 cpu‚25.196438562664483 network.recv‚777465 disk.rbd3.out‚0( disk./.usagePercent‚72.6150102583243 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875& disk.nvme0n1.out‚35954.73291015625# disk.dm-0.out‚35954.73291015625 disk.sda.in‚ 0.07421875 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 502.34375 disk.sda.out‚ 2213.1328125 disk.drbd1777.out‚0 disk.rbd2.out‚ 83.2109375* proc.memory.percent‚17.397212982177734 proc.cpu.threads‚17 network.sent‚ 3.802007e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875% proc.memory.availableMB‚ 217.59375 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda1.out‚0Ū_ñþ‚(:… ·ŒãÔ…®Ó gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚82.581' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.5162 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eà-ÿ‚(:¢ ·ŒãÔÓªßÕ% disk./.usageGB‚1082.0148658752441 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 502.34375 disk.dm-1.in‚ 0.07421875 disk.sda2.out‚0 network.recv‚783437 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0# memory_percent‚98.6745834350586 proc.cpu.threads‚17 disk.sda.out‚ 2213.78125 network.sent‚ 3.909301e+06) disk./.usagePercent‚72.61492551983046% disk.nvme0n1.out‚36058.4365234375 disk.sda1.out‚0! proc.memory.rssMB‚ 2850.359375 cpu‚25.19544878753438 disk.sda.in‚ 0.07421875 disk.rbd2.out‚ 83.2734375 disk.rbd1.out‚ 0.98291015625" disk.dm-0.out‚36058.4365234375% proc.memory.availableMB‚ 217.15625* proc.memory.percent‚17.397212982177734 disk.drbd1777.in‚0 disk.nvme0c0n1.in‚ 502.34375' disk.nvme0c0n1.out‚36058.4365234375 disk.dm-1.out‚ 2213.78125 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 2213.78125 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 502.34375 disk.sda1.in‚0elt4€‚(:© ÆŒãÔ¶¥ÒÕ disk.sda3.out‚ 2215.390625 disk.rbd0.out‚0( proc.memory.availableMB‚ 217.15234375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 2215.390625) disk./.usagePercent‚72.61491348747636% disk./.usageGB‚1082.0146865844727 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 502.34375 disk.sda.in‚ 0.07421875! proc.memory.rssMB‚ 2850.359375 network.sent‚ 3.915639e+06 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2215.390625 disk.nvme0n1.in‚ 502.34375% disk.nvme0n1.out‚36058.9638671875 cpu‚25.194466864334064 disk.rbd1.in‚0 disk.rbd1.out‚ 0.98291015625' disk.nvme0c0n1.out‚36058.9638671875 disk.rbd0.in‚0" disk.dm-0.out‚36058.9638671875 network.recv‚788171 disk.rbd2.out‚ 83.2890625* proc.memory.percent‚17.397212982177734 disk.dm-0.in‚ 502.34375 disk.sda1.in‚0 disk.drbd1777.in‚0# memory_percent‚98.6746072769165 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875. ‹‘‚(:† ÆŒãÔЋ÷Õ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5228 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Œqè?‚‚(:´ ÕŒãÔ‚†£Ö) disk./.usagePercent‚72.26997251994646 disk.rbd2.out‚83.43798828125( proc.memory.availableMB‚ 217.16015625 network.recv‚792003 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 506.39453125* proc.memory.percent‚17.397212982177734 proc.cpu.threads‚17 network.sent‚ 3.921134e+06 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚38316.537109375% disk./.usageGB‚1076.8748168945312 disk.dm-0.in‚ 506.39453125! disk.dm-0.out‚38316.537109375 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 2234.015625 disk.rbd1.out‚ 0.98291015625 disk.dm-1.in‚ 0.07421875$ disk.nvme0n1.out‚38316.537109375 cpu‚25.193481336931807 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2234.015625 disk.rbd3.out‚0! proc.memory.rssMB‚ 2850.359375 disk.rbd0.out‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 506.39453125 disk.dm-1.out‚ 2234.015625$ memory_percent‚98.67455959320068ÇHâ]œƒ‚(:‘ ÕŒãÔîãÓÖ gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚102.901' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.580199999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å!…j2„- _wandbruntime‚2207  _runtime‚2207eÁÇ…Á _runtime‚2207.484879937 _step‚163 & lr-AdamW/pg1‚0.0002197506986874286 & lr-AdamW/pg2‚0.0002197506986874286 trainer/global_step‚2049 % _timestamp‚1.7883971439250405e+09Pž"¨Ì†Æ & lr-AdamW/pg1‚0.0002197506986874286 & lr-AdamW/pg2‚0.0002197506986874286 trainer/global_step‚2049 % _timestamp‚1.7883971439250405e+09 _runtime‚2207.484879937 _step‚163£‘V 2‡- _wandbruntime‚2207  _runtime‚2207©,Ö‹È'ˆÂO ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94482363796480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77848830771712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54106919 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22428158 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38244571 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77848830771712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3730140090368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 257148928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92377672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22428160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54133099 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚81578719274496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94482082707456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73214221947904 % _timestamp‚1.7883971439258313e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36961276 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92351490 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3774365712384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77848812768768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3774108563456 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76933139390123 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76932893380919 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90707998084096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81578719274496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38244573 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90707974144000 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14532405 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77848812768768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3729906505728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3729906505728 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3730140090368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36960563 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281089024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _step‚164 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73214238987264 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81578970862080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 {ƒÍ'‰ÇO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54106919 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3718671433015 _runtime‚2207.485980311 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81578970862080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14533116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14532405 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77848812768768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3729906505728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3729906505728 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3730140090368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36960563 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281089024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _step‚164 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73214238987264 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81578970862080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94482363796480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77848830771712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54106919 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22428158 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38244571 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77848830771712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3730140090368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 257148928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92377672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22428160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54133099 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚81578719274496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94482082707456 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73214221947904 % _timestamp‚1.7883971439258313e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36961276 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92351490 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3774365712384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77848812768768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3774108563456 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76933139390123 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76932893380919 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90707998084096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81578719274496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38244573 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90707974144000 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38262750 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267621376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 _step‚165 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36979031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77885973061632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92421952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3731715654144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94527287666688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77885973061632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81617922313728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3731949252096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚729 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291561472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3720475166455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22438813 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2049 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90751353053696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76969623807735 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38262750 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54132994 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36978300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3731715654144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38262752 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81617670712832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73249165680640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76969869817147 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 _runtime‚2208.416098266 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77885955058688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90751376993792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94527579228160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3776202234368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81617922313728 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92421952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3731715654144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94527287666688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77885973061632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81617922313728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3731949252096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚729 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291561472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3720475166455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22438813 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2049 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90751353053696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76969623807735 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38262750 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54132994 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36978300 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3731715654144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38262752 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81617670712832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73249165680640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76969869817147 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 _runtime‚2208.416098266 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77885955058688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90751376993792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94527579228160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3776202234368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81617922313728 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38262750 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267621376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307¥X¨2- _wandbruntime‚2208  _runtime‚2208‚YâkTŽÎ %loss/total_loss‚61.45019414068226 0loss/FBt4/assignment_loss‚0.2607974460660579 .loss/FRt2/detection_loss‚2.097125011304068 /loss/FRt2/assignment_loss‚4.663945464860825 0loss/FRt4/assignment_loss‚2.7194849918870365 /loss/FRt3/detection_loss‚2.8369647407069287 _runtime‚2208.417054378 1loss/SRqqt2/detection_loss‚1.4778626842424274 1loss/SRqqt4/detection_loss‚0.9402147065848112 /loss/FBt4/detection_loss‚1.3280875608325005 _step‚166 /loss/FRt3/assignment_loss‚6.066827362981336 2loss/SRqqt3/assignment_loss‚2.7279443383216857 2loss/SRqqt4/assignment_loss‚2.4262472003005273 /loss/FRt4/detection_loss‚2.0410609601759444 1loss/SRqqt1/detection_loss‚3.3164066076278687 .loss/FBt2/detection_loss‚4.455493748188019 trainer/global_step‚2049 .loss/FRt1/detection_loss‚2.930873488125048 /loss/FRt1/assignment_loss‚6.126033657475522 0loss/FBt1/assignment_loss‚0.4458052425538376 /loss/FBt2/assignment_loss‚1.301072210073471 .loss/FBt1/detection_loss‚1.218208284922161 1loss/SRqqt1/assignment_loss‚3.003936482800378 2loss/SRqqt2/assignment_loss‚2.3369353830814363 .loss/FBt3/detection_loss‚3.131392425298691 % _timestamp‚1.7883971448573217e+09 0loss/FBt3/assignment_loss‚0.3917691373411799 0loss/SRqqt3/detection_loss‚3.205705004930496 epoch‚1…zâYÓ 0loss/FBt3/assignment_loss‚0.3917691373411799 0loss/SRqqt3/detection_loss‚3.205705004930496 epoch‚1 %loss/total_loss‚61.45019414068226 0loss/FBt4/assignment_loss‚0.2607974460660579 .loss/FRt2/detection_loss‚2.097125011304068 /loss/FRt2/assignment_loss‚4.663945464860825 0loss/FRt4/assignment_loss‚2.7194849918870365 /loss/FRt3/detection_loss‚2.8369647407069287 _runtime‚2208.417054378 1loss/SRqqt2/detection_loss‚1.4778626842424274 1loss/SRqqt4/detection_loss‚0.9402147065848112 /loss/FBt4/detection_loss‚1.3280875608325005 _step‚166 /loss/FRt3/assignment_loss‚6.066827362981336 2loss/SRqqt3/assignment_loss‚2.7279443383216857 2loss/SRqqt4/assignment_loss‚2.4262472003005273 /loss/FRt4/detection_loss‚2.0410609601759444 1loss/SRqqt1/detection_loss‚3.3164066076278687 .loss/FBt2/detection_loss‚4.455493748188019 trainer/global_step‚2049 .loss/FRt1/detection_loss‚2.930873488125048 /loss/FRt1/assignment_loss‚6.126033657475522 0loss/FBt1/assignment_loss‚0.4458052425538376 /loss/FBt2/assignment_loss‚1.301072210073471 .loss/FBt1/detection_loss‚1.218208284922161 1loss/SRqqt1/assignment_loss‚3.003936482800378 2loss/SRqqt2/assignment_loss‚2.3369353830814363 .loss/FBt3/detection_loss‚3.131392425298691 % _timestamp‚1.7883971448573217e+09¦š·ÿ‚(:… äŒãÔɲÆÔ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.845' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.769 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙΗT3‘‚(:¨ äŒãÔ²ñ¾Ö network.sent‚ 3.927417e+06 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0! proc.memory.rssMB‚ 2850.359375) disk./.usagePercent‚72.47659722486195% disk./.usageGB‚1079.9536743164062 disk.sda1.out‚0 disk.rbd1.out‚ 0.98291015625 disk.dm-1.out‚2236.375 disk.sda.out‚2236.375 disk.rbd2.out‚83.45361328125# proc.memory.availableMB‚217.125 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 517.3046875 disk.rbd0.in‚0$ memory_percent‚98.67477416992188 network.recv‚796560 disk.dm-0.in‚ 517.3046875 disk.rbd3.out‚0* proc.memory.percent‚17.397212982177734 disk.dm-1.in‚ 0.07421875& disk.nvme0n1.out‚42124.03662109375 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 517.3046875# disk.dm-0.out‚42124.03662109375 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚42124.03662109375 disk.sda3.in‚ 0.07421875 disk.sda3.out‚2236.375 cpu‚25.19251790598593`tš_’‚(:… óŒãÔ¯¾ûÒ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8026 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ5þ=“‚(:² óŒãÔ¦¼æÕ) disk./.usagePercent‚72.63027905968146% disk./.usageGB‚1082.2436447143555 disk.rbd2.out‚83.51611328125 disk.sda.out‚ 2237.1484375 proc.cpu.threads‚17 disk.dm-0.out‚44146.09765625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.98291015625" disk.nvme0c0n1.in‚ 564.16796875% disk.nvme0c0n1.out‚44156.84765625 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 564.16796875! proc.memory.rssMB‚ 2850.359375 network.recv‚802217 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 cpu‚25.191478411590786 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 217.109375# disk.nvme0n1.out‚44156.84765625 network.sent‚ 4.034611e+06 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 2237.1484375 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 2237.1484375* proc.memory.percent‚17.397212982177734 disk.sda1.out‚0 disk.dm-0.in‚ 564.16796875$ memory_percent‚98.67486953735352ÂFâ›”‚(: ‚ãÔ‚Ñ…Ó gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚76.396' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.279200000000001 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DÚ\{G•‚(:¼ ‚ãÔÐÖÖ" disk.nvme0c0n1.in‚ 669.41796875 disk.rbd0.in‚0% disk.nvme0n1.out‚45214.9619140625 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 669.41796875* proc.memory.percent‚17.397212982177734 network.recv‚807028 disk.dm-0.in‚ 669.41796875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 0.98291015625 disk.dm-1.in‚ 0.07421875 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2238.66015625 disk.sda3.in‚ 0.07421875' disk.nvme0c0n1.out‚45214.9619140625 disk.dm-1.out‚ 2238.66015625! proc.memory.rssMB‚ 2850.359375 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.19064838912656" disk.dm-0.out‚45214.9619140625 disk.rbd2.in‚0 disk.sda3.out‚ 2238.66015625 disk.sda1.out‚0 disk.rbd2.out‚85.10205078125$ memory_percent‚98.67336750030518 network.sent‚ 4.041002e+06) disk./.usagePercent‚72.63635693455186 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 217.35546875 proc.cpu.threads‚17% disk./.usageGB‚1082.3342094421387ŸÖ¯Ç2–- _wandbruntime‚2257  _runtime‚2257Jž·ÊÈ—Â trainer/global_step‚2099 % _timestamp‚1.7883971941946952e+09 _runtime‚ 2257.75452642 _step‚167 ' lr-AdamW/pg1‚0.00021935284574350794 ' lr-AdamW/pg2‚0.00021935284574350794³'ÛÒÍ˜Ç ' lr-AdamW/pg1‚0.00021935284574350794 ' lr-AdamW/pg2‚0.00021935284574350794 trainer/global_step‚2099 % _timestamp‚1.7883971941946952e+09 _runtime‚ 2257.75452642 _step‚167§zz"|2™- _wandbruntime‚2257  _runtime‚22578*OdšÂO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78769674322131 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 % _timestamp‚1.7883971941956973e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83526305605120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3820611924992 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96742432040448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79705945267712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820611924992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39153523 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74960556616704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3865784240640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94567542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96742727809536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37838163 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3865512411648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 trainer/global_step‚2099 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83526557192704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22960808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39153521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22960810 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14878095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.aéÂJ›dll.peak‚560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14877355 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92876943568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83526557192704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚79705945267712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295769088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55440201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚740 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92876919628800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55440201 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94593724 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39153521 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚742 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚79705927264768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3820378340352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271828992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ÃÍ'›ÇO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3820611924992 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96742432040448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79705945267712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820611924992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39153523 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74960556616704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3865784240640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94567542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96742727809536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37838163 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3865512411648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 trainer/global_step‚2099 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83526557192704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22960808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39153521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22960810 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14878095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14877355 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92876943568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83526557192704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚79705945267712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295769088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55440201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚740 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92876919628800 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55440201 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94593724 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39153521 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚742 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚79705927264768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3820378340352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271828992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78769674322131 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 % _timestamp‚1.7883971941956973e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83526305605120¨VŽ©O2œ- _wandbruntime‚2258  _runtime‚2258‘ÄúÏ&ÉM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79743087557632 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 " _timestamp‚1.7883971952454e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22971461 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83565257043456 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37855486 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3822187488768 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39171700 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79743069554688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14884755 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94638004 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94611796 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74995483310080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3822421086720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94638004 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329920 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78806404749155 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3810904399715 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55466302 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96787927453184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2099 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39171702 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92920322478592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39171700 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55440096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3822421086720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22971463 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3810675429663 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39171702 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚79743087557632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96787642183168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37856218 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92920298538496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94611796 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3867343644672 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78806158739743 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83565508644352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14884025 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55440096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3867604974592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3822187488768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74995500349440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 _runtime‚2258.806029583 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83565508644352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0~4cO«žÎM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79743087557632 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 " _timestamp‚1.7883971952454e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22971461 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83565257043456 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37855486 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3822187488768 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39171700 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79743069554688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14884755 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94638004 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94611796 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74995483310080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3822421086720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94638004 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329920 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78806404749155 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3810904399715 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55466302 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96787927453184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2099 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39171702 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92920322478592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39171700 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55440096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3822421086720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22971463 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3810675429663 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39171702 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚79743087557632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytø‹Ò) es.all.freed‚96787642183168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37856218 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92920298538496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94611796 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3867343644672 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78806158739743 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83565508644352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14884025 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55440096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3867604974592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3822187488768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74995500349440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 _runtime‚2258.806029583 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83565508644352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0©"ù02Ÿ- _wandbruntime‚2258  _runtime‚2258òI´€T Î 1loss/SRqqt2/assignment_loss‚2.401683066572462 2loss/SRqqt3/assignment_loss‚2.6957749644915263 2loss/SRqqt4/assignment_loss‚2.7039433070171026 0loss/FBt1/assignment_loss‚0.4448577647315324 .loss/FBt2/detection_loss‚4.153702914714813 trainer/global_step‚2099 /loss/FRt2/assignment_loss‚4.119498297816417 .loss/FRt1/detection_loss‚2.881061961776332 1loss/SRqqt2/detection_loss‚1.7131390339798398 $ _timestamp‚1.788397195247627e+09 1loss/SRqqt3/detection_loss‚2.7000503559907276 /loss/FBt1/detection_loss‚1.5058158059883864 %loss/total_loss‚60.14404507686602 epoch‚1 /loss/FRt1/assignment_loss‚6.195492694252415 0loss/FRt4/assignment_loss‚2.5792137538564615 0loss/FBt3/assignment_loss‚0.4122082330094372 /loss/FRt2/detection_loss‚2.0940860548957447 /loss/FBt4/detection_loss‚1.4757098953884382 1loss/FBt2/assignment_loss‚0.24775408926507225 /loss/FBt4/assignment_loss‚0.320810521522407 .loss/FBt3/detection_loss‚4.310849387198687 _step‚170 /loss/FRt3/assignment_loss‚5.969651182492574 2loss/SRqqt1/assignment_loss‚2.6559000743760004 0loss/SRqqt1/detection_loss‚3.016527333193355 1loss/SRqqt4/detection_loss‚1.0755474222479042 -loss/FRt3/detection_loss‚2.65960331261158 /loss/FRt4/detection_loss‚1.8111636494768077 _runtime‚2258.807390197–×D%Y¡Ó /loss/FRt2/assignment_loss‚4.119498297816417 .loss/FRt1/detection_loss‚2.881061961776332 1loss/SRqqt2/detection_loss‚1.7131390339798398 $ _timestamp‚1.788397195247627e+09 1loss/SRqqt3/detection_loss‚2.7000503559907276 /loss/FBt1/detection_loss‚1.5058158059883864 %loss/total_loss‚60.14404507686602 epoch‚1 /loss/FRt1/assignment_loss‚6.195492694252415 0loss/FRt4/assignment_loss‚2.5792137538564615 0loss/FBt3/assignment_loss‚0.4122082330094372 /loss/FRt2/detection_loss‚2.0940860548957447 /loss/FBt4/detection_loss‚1.4757098953884382 1loss/FBt2/assignment_loss‚0.24775408926507225 /loss/FBt4/assignment_loss‚0.320810521522407 .loss/FBt3/detection_loss‚4.310849387198687 _step‚170 /loss/FRt3/assignment_loss‚5.969651182492574 2loss/SRqqt1/assignment_loss‚2.6559000743760004 0loss/SRqqt1/detection_loss‚3.016527333193355 1loss/SRqqt4/detection_loss‚1.0755474222479042 -loss/FRt3/detection_loss‚2.65960331261158 /loss/FRt4/detection_loss‚1.8111636494768077 _runtime‚2258.807390197 1loss/SRqqt2/assignment_loss‚2.401683066572462 2loss/SRqqt3/assignment_loss‚2.6957749644915263 2loss/SRqqt4/assignment_loss‚2.7039433070171026 0loss/FBt1/assignment_loss‚0.4448577647315324 .loss/FBt2/detection_loss‚4.153702914714813 trainer/global_step‚2099ª>»Šê¢‚(:… ‘ãÔõÇúÒ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9418 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±)TÐB£‚(:· ‘ãÔ®ª¹Õ disk.rbd0.in‚0 disk.nvme0n1.in‚ 669.41796875 disk.dm-0.in‚ 669.41796875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda3.in‚ 0.07421875 disk.sda.in‚ 0.07421875! proc.memory.rssMB‚ 2850.359375% disk./.usageGB‚1082.3342475891113" disk.nvme0c0n1.in‚ 669.41796875 network.sent‚ 4.046348e+06 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 2239.859375 disk.dm-1.in‚ 0.07421875& proc.memory.availableMB‚ 217.046875 disk.sda2.out‚0 disk.rbd1.out‚ 0.98291015625 disk.rbd0.out‚0# disk.dm-0.out‚45215.34521484375 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd2.out‚85.16455078125 disk.rbd3.out‚0 disk.sda3.out‚ 2239.859375( disk.nvme0c0n1.out‚45215.34521484375( disk./.usagePercent‚72.6363594946272 disk.sda2.in‚0& disk.nvme0n1.out‚45215.34521484375 disk.sda.out‚ 2239.859375$ memory_percent‚98.67525100708008* proc.memory.percent‚17.397212982177734 cpu‚25.189743220015018 network.recv‚810794 disk.sda1.out‚0 disk.rbd1.in‚0Ð OD›¤‚(:  ãÔ¨ŠÿÓ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.470599999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÀÈG¥‚(:¼  ãÔô¦«Ö) disk./.usagePercent‚72.63214509859853 disk.rbd2.out‚85.18017578125 disk.rbd3.in‚0 disk.sda3.out‚ 2252.7109375 disk.nvme0n1.in‚ 669.41796875$ memory_percent‚98.67537021636963 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚45552.52587890625 network.sent‚ 4.154558e+06 disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 669.41796875 disk.sda.in‚ 0.07421875 network.recv‚817195( disk.nvme0c0n1.out‚45552.52587890625 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.98291015625" disk.nvme0c0n1.in‚ 669.41796875 disk.rbd0.in‚0 disk.sda.out‚ 2252.7109375# disk.dm-0.out‚45552.52587890625( proc.memory.availableMB‚ 217.02734375 disk.sda1.out‚0 disk.dm-1.out‚ 2252.7109375* proc.memory.percent‚17.397212982177734% disk./.usageGB‚1082.2714500427246 disk.sda1.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.out‚0! proc.memory.rssMB‚ 2850.359375 cpu‚25.18881414695247÷º„¦‚(:… ¯ãÔã¹êÒ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚81.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.2458 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e"$F§‚(:» ¯ãÔçªÛÕ$ memory_percent‚98.67382049560547% proc.memory.availableMB‚ 217.28125( disk.nvme0c0n1.out‚45558.18994140625! proc.memory.rssMB‚ 2850.359375 proc.cpu.threads‚17 disk.rbd2.out‚ 85.2666015625 disk.dm-1.in‚ 0.07421875$ disk./.usageGB‚1082.265235900879 disk.rbd2.in‚0 disk.sda3.out‚ 2289.56640625* proc.memory.percent‚17.397212982177734 network.sent‚ 4.159847e+06 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 669.41796875 disk.dm-0.in‚ 669.41796875 network.recv‚820866 disk.dm-1.out‚ 2289.56640625 cpu‚25.188039115480795) disk./.usagePercent‚72.63172806232527 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda.out‚ 2289.56640625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚45558.18994140625# disk.dm-0.out‚45558.18994140625 disk.sda.in‚ 0.07421875 disk.rbd3.out‚0 disk.rbd1.out‚ 0.98291015625" disk.nvme0c0n1.in‚ 669.41796875 disk.rbd0.in‚0‡µÏ¨‚(:„ ¾ãÔ€Æ¨Ó gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.259 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Ó/wD©‚(:¹ ¾ãÔ¶ï¾Õ disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚72.63178873611086" disk.nvme0c0n1.in‚ 669.41796875 disk.dm-1.out‚ 2290.99609375 disk.rbd2.in‚0' proc.memory.availableMB‚ 217.2734375 disk.nvme0n1.in‚ 669.41796875% disk.nvme0n1.out‚45560.2060546875 disk.sda.in‚ 0.07421875 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 2290.99609375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚45560.2060546875 disk.dm-1.in‚ 0.07421875 network.sent‚ 4.166089e+06 disk.dm-0.in‚ 669.41796875 disk.rbd2.out‚ 85.2822265625 disk.rbd3.out‚0 disk.rbd1.out‚ 0.98291015625* proc.memory.percent‚17.397212982177734 disk.sda.out‚ 2290.99609375 disk.sda1.out‚0$ memory_percent‚98.67386817932129 cpu‚25.1872649474043% disk./.usageGB‚1082.2661399841309 disk.drbd1777.in‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2850.359375 network.recv‚825545" disk.dm-0.out‚45560.2060546875`˜5I2ª- _wandbruntime‚2311  _runtime‚2311Àª(àÉ«Ã % _timestamp‚1.7883972480726151e+09 _runtime‚2311.632477314 _step‚171 ' lr-AdamW/pg1‚0.00021893251789163954 ' lr-AdamW/pg2‚0.00021893251789163954 trainer/global_step‚2149£OLÎ¬È _runtime‚2311.632477314 _step‚171 ' lr-AdamW/pg1‚0.00021893251789163954 ' lr-AdamW/pg2‚0.00021893251789163954 trainer/global_step‚2149 % _timestamp‚1.7883972480726151e+09«›Y_S2­- _wandbruntime‚2311  _runtime‚23111Ÿ1™È'®ÂO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81563041760768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3911083759616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911083759616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3899071959431 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96809776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56721123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38715519 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85473891935744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3910850174976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 % _timestamp‚1.7883972480736291e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23493458 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80605963244935 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99003075771392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3956925374464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3899300929275 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96783594 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38714812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 _step‚172 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15221354 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95045889053696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15222059 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95045865113600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85474143523328 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85473891935744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40062471 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 261343232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76706908324864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56721123 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85474143523328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 285283328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2311.633933448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3910850174976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56747303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40062471 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3957186717696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 trainer/global_step‚2149 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56747303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40062473 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Ÿ÷×Í'¯ÇO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81563041760768 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2311.633933448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3910850174976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56747303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40062471 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3957186717696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 trainer/global_step‚2149 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56747303 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40062473 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81563041760768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3911083759616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911083759616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3899071959431 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96809776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56721123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38715519 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85473891935744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3910850174976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 % _timestamp‚1.7883972480736291e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23493458 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80605963244935 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99003075771392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3956925374464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3899300929275 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96783594 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38714812 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 _step‚172 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15221354 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95045889053696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15222059 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95045865113600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85474143523328 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85473891935744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40062471 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 261343232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76706908324864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56721123 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85474143523328 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 285283328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3912659323392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96854056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81600202053632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40080652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95089244023296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3958751352832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 % _timestamp‚1.7883972491241775e+09 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23504111 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56747198 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96827848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15228753 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2149 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2312.684680946 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56773404 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95089267963392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81600202053632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85513094974976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3912892921344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76741835018240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81600184050688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81600184050688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80642939681163 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38732157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23504113 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40080650 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2V¿]ø×&²ÑM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95089244023296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3958751352832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 % _timestamp‚1.7883972491241775e+09 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23504111 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56747198 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96827848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15228753 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2149 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2312.684680946 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56773404 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95089267963392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81600202053632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85513094974976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3912892921344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76741835018240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81600184050688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81600184050688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80642939681163 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38732157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23504113 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40080650 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80642693671751 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76741817978880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15228046 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99047995376128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3901104662923 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3912659323392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96854056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81600202053632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40080652 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360­`•É2³- _wandbruntime‚2312  _runtime‚2312µ7ZU´Ï 1loss/SRqqt3/assignment_loss‚3.039274628673281 trainer/global_step‚2149 /loss/FRt3/detection_loss‚2.7247540201689744 % _timestamp‚1.7883972491261737e+09 _runtime‚ 2312.68589849 /loss/FRt1/assignment_loss‚6.028200010458629 0loss/FRt3/assignment_loss‚5.9343979294235645 /loss/FRt1/detection_loss‚2.5011962354183197 .loss/FRt4/detection_loss‚2.043748835633908 1loss/SRqqt2/detection_loss‚1.5299566130932063 1loss/SRqqt4/detection_loss‚1.1956396405471534 /loss/FRt4/assignment_loss‚2.899227125304086 2loss/SRqqt2/assignment_loss‚2.4545824650215775 2loss/SRqqt4/assignment_loss‚2.6308959424495697 1loss/FBt1/assignment_loss‚0.39590616719900373 0loss/FBt4/assignment_loss‚0.2307850647369681 1loss/SRqqt1/assignment_loss‚2.903952366113663 /loss/FBt1/detection_loss‚1.4181093150449944 .loss/FBt3/detection_loss‚3.107754661486699 /loss/FBt4/detection_loss‚1.5925782670577366 .loss/FRt2/assignment_loss‚4.51268430341754 0loss/FBt3/assignment_loss‚0.5287168460748105 1loss/SRqqt3/detection_loss‚2.5726214924028943 epoch‚1 _step‚174 0loss/FBt2/assignment_loss‚0.4631169974803925 .loss/FRt2/detection_loss‚2.186449903145171 0loss/SRqqt1/detection_loss‚3.601516753435135 .loss/FBt2/detection_loss‚3.267739677429199 &loss/total_loss‚59.763805261216476¸S™iZµÔ /loss/FRt3/detection_loss‚2.7247540201689744 % _timestamp‚1.7883972491261737e+09 _runtime‚ 2312.68589849 /loss/FRt1/assignment_loss‚6.028200010458629 0loss/FRt3/assignment_loss‚5.9343979294235645 /loss/FRt1/detection_loss‚2.5011962354183197 .loss/FRt4/detection_loss‚2.043748835633908 1loss/SRqqt2/detection_loss‚1.5299566130932063 1loss/SRqqt4/detection_loss‚1.1956396405471534 /loss/FRt4/assignment_loss‚2.899227125304086 2loss/SRqqt2/assignment_loss‚2.4545824650215775 2loss/SRqqt4/assignment_loss‚2.6308959424495697 1loss/FBt1/assignment_loss‚0.39590616719900373 0loss/FBt4/assignment_loss‚0.2307850647369681 1loss/SRqqt1/assignment_loss‚2.903952366113663 /loss/FBt1/detection_loss‚1.4181093150449944 .loss/FBt3/detection_loss‚3.107754661486699 /loss/FBt4/detection_loss‚1.5925782670577366 .loss/FRt2/assignment_loss‚4.51268430341754 0loss/FBt3/assignment_loss‚0.5287168460748105 1loss/SRqqt3/detection_loss‚2.5726214924028943 epoch‚1 _step‚174 0loss/FBt2/assignment_loss‚0.4631169974803925 .loss/FRt2/detection_loss‚2.186449903145171 0loss/SRqqt1/detection_loss‚3.601516753435135 .loss/FBt2/detection_loss‚3.267739677429199 &loss/total_loss‚59.763805261216476 1loss/SRqqt3/assignment_loss‚3.039274628673281 trainer/global_step‚2149®©Øz6¶‚(:… ÍãÔ«Ö‰Ó gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÌ3:·‚(:¯ ÍãÔé„áÖ% disk.nvme0n1.out‚45659.3310546875 disk.sda3.out‚ 2309.5625) disk./.usagePercent‚72.63178873611086% disk./.usageGB‚1082.2661399841309' disk.nvme0c0n1.out‚45659.3310546875 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2309.5625 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 4.171573e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 669.41796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0$ memory_percent‚98.67556095123291 disk.rbd0.out‚0 disk.nvme0n1.in‚ 669.41796875" disk.dm-0.out‚45659.3310546875" disk.nvme0c0n1.in‚ 669.41796875 disk.sda.out‚ 2309.5625 disk.sda3.in‚ 0.07421875 disk.rbd2.in‚0( proc.memory.availableMB‚ 216.99609375 cpu‚25.18636163169843 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.06884765625 network.recv‚829496 disk.rbd2.out‚ 86.9814453125 disk.sda2.in‚0! proc.memory.rssMB‚ 2850.359375* proc.memory.percent‚17.397212982177734E©á’¸‚(:„ ÜãÔ¦×ÄÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.259 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üGAHG¹‚(:¼ ÜãÔƒ‰ÑÖ" disk.nvme0c0n1.in‚ 669.41796875( proc.memory.availableMB‚ 216.98828125 disk.sda1.in‚0 disk.sda3.out‚ 2310.77734375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 669.41796875 disk.sda.in‚ 0.07421875! proc.memory.rssMB‚ 2850.359375 cpu‚25.185499457131385 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 86.9970703125 disk.rbd3.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚45664.8408203125" disk.dm-0.out‚45664.8408203125$ memory_percent‚98.67560863494873) disk./.usagePercent‚72.63177977584718* proc.memory.percent‚17.397212982177734% disk./.usageGB‚1082.2660064697266 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1.06884765625 disk.dm-1.out‚ 2310.77734375 disk.sda.out‚ 2310.77734375 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 4.279746e+06 network.recv‚836356 disk.sda3.in‚ 0.07421875% disk.nvme0n1.out‚45664.8408203125 disk.rbd1.in‚0 disk.dm-0.in‚ 669.41796875‹øÐ ›º‚(: ëãÔÜõ–Ó gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚89.829' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.965799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 X ×F»‚(:» ëãÔš¡üÕ disk.nvme0n1.in‚ 669.41796875# disk.dm-0.out‚45764.28955078125 disk.sda.in‚ 0.07421875 disk.sda3.out‚ 2313.24609375& disk.nvme0n1.out‚45764.28955078125! proc.memory.rssMB‚ 2850.359375 disk.dm-1.in‚ 0.07421875 disk.drbd1777.in‚0& proc.memory.availableMB‚ 216.984375 disk.sda2.out‚0 disk.rbd1.out‚ 1.13134765625 disk.drbd1777.out‚0 disk.dm-1.out‚ 2313.24609375 disk.sda.out‚ 2313.24609375 disk.rbd3.in‚0( disk.nvme0c0n1.out‚45764.28955078125 network.recv‚840280$ disk./.usageGB‚1080.939296722412 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0$ memory_percent‚98.67563247680664 network.sent‚ 4.284948e+06 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 669.41796875* proc.memory.percent‚17.397212982177734 cpu‚25.18476588193005) disk./.usagePercent‚72.54274317152128 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 669.41796875 disk.rbd2.out‚ 87.1220703125 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0Z‰>2¼- _wandbruntime‚2366  _runtime‚2366|Õùɽà ' lr-AdamW/pg1‚0.00021848980272751777 ' lr-AdamW/pg2‚0.00021848980272751777 trainer/global_step‚2199 % _timestamp‚1.7883973027092624e+09 _runtime‚2366.269116597 _step‚175B™q‰Î¾È ' lr-AdamW/pg1‚0.00021848980272751777 ' lr-AdamW/pg2‚0.00021848980272751777 trainer/global_step‚2199 % _timestamp‚1.7883973027092624e+09 _runtime‚2366.269116597 _step‚175¯.þ§W2¿- _wandbruntime‚2366  _runtime‚2366$z–×ÀÄO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83420156256768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚730 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78453242993664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4001322009600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 _runtime‚2366.270240575 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83420156256768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 % _timestamp‚1.7883973027101343e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚87421478266368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97214810598400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚87421729853952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15566593 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58054405 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39592703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 E=DeviceStatsMonitor.on_train_batch_êu(óstart/segment.all.allocated‚560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58028225 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97214834538496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3989501192483 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58028225 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚87421729853952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40971423 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39591973 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚2199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40971421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4001555594240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 285283328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4048407059968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚82442498176943 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87421478266368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99025828 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58054405 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83420174259712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82442744186147 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87421478266368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99025828 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58054405 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83420174259712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82442744186147 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 _runtime‚2366.270240575 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83420156256768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 % _timestamp‚1.7883973027101343e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚87421478266368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97214810598400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚87421729853952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15566593 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58054405 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39592703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58028225 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97214834538496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3989501192483 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58028225 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚87421729853952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40971423 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844°VF2Â- _wandbruntime‚2367  _runtime‚2367ç±ñqÔ&ÃÎM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40989602 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99070108 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99070108 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 % _timestamp‚1.7883973037469947e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚2367.308044423 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83457298546688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99043900 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58054300 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39610213 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83457316549632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101308708035072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40989600 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87460681305600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15572741 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87460681305600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39609502 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4050233256960 trainer/global_step‚2199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58080506 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83457298546688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4003131158016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15573450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3991304926131 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24036763 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83457316549632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58054300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4003364755968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97258189508096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚177 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4003131158016 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4003364755968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97258189508096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚177 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4003131158016 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99070108 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99070108 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 % _timestamp‚1.7883973037469947e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚2367.308044423 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83457298546688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99043900 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58054300 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39610213 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83457316549632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101308708035072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40989600 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87460681305600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15572741 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87460681305600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39609502 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4050233256960 trainer/global_step‚2199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58080506 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83457298546688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4003131158016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15573450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3991304926131 REDeviceStatsMonitor.on_train‹ŸÚÄ_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24036763 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560±vÙ<\2Å- _wandbruntime‚2367  _runtime‚2367Â5SÆÍ 1loss/FBt4/assignment_loss‚0.20228764106971595 .loss/FBt3/detection_loss‚2.486639908381871 _step‚178 0loss/SRqqt3/assignment_loss‚2.62873317366061 .loss/FRt1/detection_loss‚2.445951819419861 0loss/SRqqt1/detection_loss‚3.457705408334732 0loss/SRqqt4/detection_loss‚1.211594572601219 /loss/FRt2/detection_loss‚2.0865046063722192 /loss/FBt3/assignment_loss‚0.762574407570615 -loss/FBt2/detection_loss‚4.06561678647995 _runtime‚2367.309437909 0loss/FRt4/assignment_loss‚3.1034408304198036 /loss/FBt1/detection_loss‚1.2015808998009885 /loss/FRt3/assignment_loss‚5.818916376899271 2loss/SRqqt4/assignment_loss‚2.4501466866996555 1loss/SRqqt2/detection_loss‚1.5490181504131912 1loss/SRqqt3/detection_loss‚2.8009112088576607 &loss/total_loss‚59.683559383092046 1loss/SRqqt1/assignment_loss‚2.821805020173391 $ _timestamp‚1.788397303749663e+09 /loss/FRt1/assignment_loss‚6.399177447609279 2loss/SRqqt2/assignment_loss‚2.4755580494930216 0loss/FBt1/assignment_loss‚0.3544645543707337 /loss/FRt3/detection_loss‚2.7260466099661937 /loss/FBt4/detection_loss‚1.6179519538976708 epoch‚1 trainer/global_step‚2199 /loss/FRt2/assignment_loss‚4.601907188609495 /loss/FRt4/detection_loss‚2.0210827840813277 0loss/FBt2/assignment_loss‚0.3939432979095727DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84279033108951 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101241880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚745 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 297866240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103523543296512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4091793844224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85277270752768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚743 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4140061139456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24558758 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4079472485847 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15911500 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59335327 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89369316184576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41880371 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99383780023296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101215698 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41880373 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4091793844224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 _step‚180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41880373 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85277288755712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7883973567474353e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85277270752768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41880371 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15912243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103523841162752 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89369064596992 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚89369316184576 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560؇úÍÓ'ÕÍO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4091793844224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 _step‚180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41880373 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85277288755712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7883973567474353e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85277270752768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41880371 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15912243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103523841162752 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89369064596992 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚89369316184576 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85277288755712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24558760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84279033108951 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101241880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚745 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 297866240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103523543296512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4091793844224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85277270752768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚743 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4140061139456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24558758 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4079472485847 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15911500 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59335327 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89369316184576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41880371 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99383780023296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101215698 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41880373´å£Yõ2Ö- _wandbruntime‚2421  _runtime‚2421‹#ÔO«×ÒM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101286160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15918296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 >6DeviceStatsMonitor.on_tr¯¼A-$ain_batch_end/num_device_alloc‚560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4093836590592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59387608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41898550 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84316009545179 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40488437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41898550 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101286160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89408267636224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41898552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103569039276544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99427134992896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59387608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103568749812224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80234487316480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99427158932992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4141614819328 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85314431045632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4093602992640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80234504355840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59361402 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40487707 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24569411 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4141880343552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85314431045632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4081276219287 _step‚181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85314413042688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101259952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4081505189339 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41898552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚730 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15919024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59361402 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84315763535767 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0h’äÈÝ&Ø×M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4081505189339 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41898552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚730 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15919024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59361402 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84315763535767 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101286160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15918296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4093836590592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59387608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41898550 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84316009545179 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40488437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41898550 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101286160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89408267636224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41898552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103569039276544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99427134992896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59387608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103568749812224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80234487316480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99427158932992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4141614819328 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85314431045632 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4093602992640 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80234504355840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59361402 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40487707 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24569411 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4141880343552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85314431045632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4081276219287 _step‚181 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85314413042688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101259952µÔN2Ù- _wandbruntime‚2421  _runtime‚2421AämWÚÑ 1loss/SRqqt4/detection_loss‚0.8333026941120625 %loss/total_loss‚59.68514022049808 0loss/SRqqt3/assignment_loss‚2.70583838224411 1loss/SRqqt3/detection_loss‚2.6708996202796698 /loss/FBt4/detection_loss‚1.3083813355324116 0loss/FBt3/assignment_loss‚0.8135336735150693 /loss/FRt1/detection_loss‚2.3805102109909058 1loss/SRqqt2/detection_loss‚1.6913799430007366 /loss/FBt1/detection_loss‚1.5728315922295295 .loss/FBt3/detection_loss‚3.718985523496355 epoch‚1 _step‚182 /loss/FRt2/assignment_loss‚4.225088323865618 0loss/FRt4/assignment_loss‚3.0287669506939974 0loss/FRt3/assignment_loss‚5.9336700439453125 2loss/SRqqt2/assignment_loss‚2.5231906736074987 /loss/FRt2/detection_loss‚2.4926633962563107 .loss/FRt4/detection_loss‚1.791540110653097 0loss/SRqqt1/detection_loss‚3.229725354247623 _runtime‚2421.297874447 /loss/FRt1/assignment_loss‚6.323535828363328 .loss/FRt3/detection_loss‚2.310388663820192 .loss/FBt2/detection_loss‚3.306761682033539 0loss/FBt1/assignment_loss‚0.4152882908361562 trainer/global_step‚2249 % _timestamp‚1.7883973577380426e+09 2loss/SRqqt1/assignment_loss‚3.0730681551827326 2loss/SRqqt4/assignment_loss‚2.7320072555541994 1loss/FBt2/assignment_loss‚0.31225886568427086 0loss/FBt4/assignment_loss‚0.2915236503533495é>«\ÛÖ 0loss/FRt3/assignment_loss‚5.9336700439453125 2loss/SRqqt2/assignment_loss‚2.5231906736074987 /loss/FRt2/detection_loss‚2.4926633962563107 .loss/FRt4/detection_loss‚1.791540110653097 0loss/SRqqt1/detection_loss‚3.229725354247623 _runtime‚2421.297874447 /loss/FRt1/assignment_loss‚6.323535828363328 .loss/FRt3/detection_loss‚2.310388663820192 .loss/FBt2/detection_loss‚3.306761682033539 0loss/FBt1/assignment_loss‚0.4152882908361562 trainer/global_step‚2249 % _timestamp‚1.7883973577380426e+09 2loss/SRqqt1/assignment_loss‚3.0730681551827326 2loss/SRqqt4/assignment_loss‚2.7320072555541994 1loss/FBt2/assignment_loss‚0.31225886568427086 0loss/FBt4/assignment_loss‚0.2915236503533495 1loss/SRqqt4/detection_loss‚0.8333026941120625 %loss/total_loss‚59.68514022049808 0loss/SRqqt3/assignment_loss‚2.70583838224411 1loss/SRqqt3/detection_loss‚2.6708996202796698 /loss/FBt4/detection_loss‚1.3083813355324116 0loss/FBt3/assignment_loss‚0.8135336735150693 /loss/FRt1/detection_loss‚2.3805102109909058 1loss/SRqqt2/detection_loss‚1.6913799430007366 /loss/FBt1/detection_loss‚1.5728315922295295 .loss/FBt3/detection_loss‚3.718985523496355 epoch‚1 _step‚182 /loss/FRt2/assignment_loss‚4.225088323865618 0loss/FRt4/assignment_loss‚3.0287669506939974¶¾Î4 ›Ü‚(: ¶ŽãÔŒñíÒ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚75.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.133600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¥«±FÝ‚(:» ¶ŽãÔ“»ØÕ% disk./.usageGB‚1080.9423713684082 disk.rbd2.out‚89.01708984375" disk.nvme0c0n1.in‚ 669.41796875 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2850.36328125* proc.memory.percent‚17.397236824035645 cpu‚25.180860715495825 proc.cpu.threads‚17) disk./.usagePercent‚72.54294951359388 disk.rbd3.in‚0" disk.dm-0.out‚45972.2958984375$ memory_percent‚98.67632389068604( proc.memory.availableMB‚ 216.87109375 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 2335.34765625 disk.rbd0.out‚0 disk.dm-0.in‚ 669.41796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 network.sent‚ 4.415715e+06 disk.sda3.out‚ 2335.34765625 network.recv‚862864 disk.rbd3.out‚0' disk.nvme0c0n1.out‚45972.2958984375 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 669.41796875 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2335.34765625 disk.rbd2.in‚0 disk.rbd1.out‚ 1.2265625% disk.nvme0n1.out‚45972.2958984375aqC›Þ‚(: ÅŽãÔé•ìÔ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚96.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.261799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â<¶ÂEß‚(:º ÅŽãÔçäéÖ disk.rbd1.out‚ 1.28955078125 disk.rbd2.out‚ 89.103515625 disk.sda3.in‚ 0.07421875' proc.memory.availableMB‚ 217.1171875 cpu‚25.180042413753572 network.recv‚868747 disk.sda2.in‚0 disk.sda.out‚ 2337.41015625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0# memory_percent‚98.6748218536377* proc.memory.percent‚17.397260665893555) disk./.usagePercent‚72.54295207366923 disk.dm-1.out‚ 2337.41015625" proc.memory.rssMB‚ 2850.3671875 network.sent‚ 4.523175e+06 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0' disk.nvme0c0n1.out‚45972.6044921875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda3.out‚ 2337.41015625" disk.nvme0c0n1.in‚ 669.41796875 disk.nvme0n1.in‚ 669.41796875% disk.nvme0n1.out‚45972.6044921875 disk.sda1.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1080.9424095153809 disk.dm-0.in‚ 669.41796875" disk.dm-0.out‚45972.6044921875 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0Jê •à‚(:… ÔŽãÔÅó¤Ó gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2954 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü9pîAá‚(:¶ ÔŽãÔŸ¼× disk.sda3.out‚ 2352.84765625" disk.nvme0c0n1.in‚ 669.43359375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 669.43359375# disk.nvme0n1.out‚45973.08984375 disk.sda.out‚ 2352.84765625 disk.rbd2.in‚0$ memory_percent‚98.67384433746338) disk./.usagePercent‚72.54179107950144 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2850.3671875* proc.memory.percent‚17.397260665893555 network.sent‚ 4.529423e+06 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.28955078125% disk.nvme0c0n1.out‚45973.08984375 disk.dm-1.in‚ 0.07421875( proc.memory.availableMB‚ 217.27734375 disk.sda.in‚ 0.07421875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 2352.84765625 disk.rbd0.in‚0 disk.dm-0.in‚ 669.43359375 cpu‚25.179345017161182 network.recv‚873457% disk./.usageGB‚1080.9251098632812 disk.rbd2.out‚ 89.119140625 disk.rbd1.in‚0 disk.dm-0.out‚45973.08984375\ ¯|2â- _wandbruntime‚2474  _runtime‚2474§i&šÉãà ' lr-AdamW/pg1‚0.00021753758415347599 ' lr-AdamW/pg2‚0.00021753758415347599 trainer/global_step‚2299 % _timestamp‚1.7883974108071508e+09 _runtime‚2474.367253155 _step‚183f+–ÎäÈ _runtime‚2474.367253155 _step‚183 ' lr-AdamW/pg1‚0.00021753758415347599 ' lr-AdamW/pg2‚0.00021753758415347599 trainer/global_step‚2299 % _timestamp‚1.7883974108071508e+09·§áÅf2å- _wandbruntime‚2474  _runtime‚2474aUwŽæÊO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101552725508096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86115568040959 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91316650927616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16255008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4169672749055 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4182265678848 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91316902515200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293671936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86115814050163 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42789323 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4169901718899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4182499263488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 % _timestamp‚1.7883974108079157e‹;¥¼+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87134403251712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87134385248768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42789321 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4231224160768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 _step‚184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41347149 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16255739 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81945912331264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91316902515200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105783925728768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60668609ã Û·Õ'çÏO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25091408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101552701568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4231493892608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42789323 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91316902515200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105783925728768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60668609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101552725508096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86115568040959 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91316650927616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16255008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4169672749055 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4182265678848 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91316902515200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293671936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86115814050163 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42789323 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4169901718899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4182499263488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 % _timestamp‚1.7883974108079157e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87134403251712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87134385248768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42789321 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4231224160768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 _step‚184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41347149 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16255739 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81945912331264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103502212 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60668504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4171705452547 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91355853966848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4184308425216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4233332440064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81980821985280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87171527538688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60694710 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25102063 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91355602365952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87171527538688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42807500 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4184308425216 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4171476482495 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚185 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105829436857856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86152298467775 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103502212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 _runtime‚2475.404074827 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86152544477187 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4184074827264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16262118 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41364179 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103476004 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42807500 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81980839024640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42807502 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25102061 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287367168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105829149490688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Eh­²ZìÙM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105829436857856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86152298467775 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103502212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 _runtime‚2475.404074827 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86152544477187 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4184074827264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16262118 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41364179 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103476004 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42807500 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81980839024640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42807502 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25102061 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287367168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105829149490688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103502212 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60668504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4171705452547 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91355853966848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4184308425216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4233332440064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81980821985280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87171527538688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60694710 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25102063 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91355602365952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87171527538688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42807500 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4184308425216 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4171476482495 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚185 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307¹i‡BÕ2í- _wandbruntime‚2475  _runtime‚2475¬a¸žTîÎ %loss/total_loss‚61.22719031378386 % _timestamp‚1.7883974118457212e+09 1loss/SRqqt2/assignment_loss‚2.275968749229222 /loss/FRt3/detection_loss‚2.5179926622245046 /loss/FBt4/detection_loss‚1.4393399426868807 .loss/FRt2/assignment_loss‚4.51115327314897 0loss/FBt1/assignment_loss‚0.5506871497644852 0loss/FBt3/assignment_loss‚0.4682062568214557 1loss/FBt4/assignment_loss‚0.25247769020340155 2loss/SRqqt1/assignment_loss‚2.9930857467651366 /loss/FRt2/detection_loss‚2.1660458033735104 1loss/SRqqt1/detection_loss‚3.0994424390792847 trainer/global_step‚2299 -loss/FRt1/assignment_loss‚6.4053294217145 /loss/FRt3/assignment_loss‚5.889108000861274 0loss/FBt2/assignment_loss‚1.2805752605199814 _step‚186 2loss/SRqqt4/assignment_loss‚2.1999411994218825 /loss/FRt1/detection_loss‚3.1789399716589184 1loss/SRqqt2/detection_loss‚1.7248522290219999 .loss/FBt1/detection_loss‚1.561938312825081 .loss/FBt3/detection_loss‚2.936330338319143 /loss/FRt4/assignment_loss‚2.693025991651747 2loss/SRqqt3/assignment_loss‚3.0020258824030557 -loss/FRt4/detection_loss‚2.02658767302831 1loss/SRqqt3/detection_loss‚2.8663451145092647 _runtime‚2475.405477811 1loss/SRqqt4/detection_loss‚0.9621843656897545 /loss/FBt2/detection_loss‚4.2256068388621015 epoch‚1Cû¡=YïÓ 1loss/SRqqt2/detection_loss‚1.7248522290219999 .loss/FBt1/detection_loss‚1.561938312825081 .loss/FBt3/detection_loss‚2.936330338319143 /loss/FRt4/assignment_loss‚2.693025991651747 2loss/SRqqt3/assignment_loss‚3.0020258824030557 -loss/FRt4/detection_loss‚2.02658767302831 1loss/SRqqt3/detection_loss‚2.8663451145092647 _runtime‚2475.405477811 1loss/SRqqt4/detection_loss‚0.9621843656897545 /loss/FBt2/detection_loss‚4.2256068388621015 epoch‚1 %loss/total_loss‚61.22719031378386 % _timestamp‚1.7883974118457212e+09 1loss/SRqqt2/assignment_loss‚2.275968749229222 /loss/FRt3/detection_loss‚2.5179926622245046 /loss/FBt4/detection_loss‚1.4393399426868807 .loss/FRt2/assignment_loss‚4.51115327314897 0loss/FBt1/assignment_loss‚0.5506871497644852 0loss/FBt3/assignment_loss‚0.4682062568214557 1loss/FBt4/assignment_loss‚0.25247769020340155 2loss/SRqqt1/assignment_loss‚2.9930857467651366 /loss/FRt2/detection_loss‚2.1660458033735104 1loss/SRqqt1/detection_loss‚3.0994424390792847 trainer/global_step‚2299 -loss/FRt1/assignment_loss‚6.4053294217145 /loss/FRt3/assignment_loss‚5.889108000861274 0loss/FBt2/assignment_loss‚1.2805752605199814 _step‚186 2loss/SRqqt4/assignment_loss‚2.1999411994218825 /loss/FRt1/detection_loss‚3.1789399716589184ºÀD8Êð‚(:„ òŽãÔßì¢Ó gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.069' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6138 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÊicHñ‚(:½ òŽãÔ–ÈÍÖ network.sent‚ 4.631583e+06) disk./.usagePercent‚72.54178877543363 disk.sda.out‚ 2375.99609375 disk.drbd1777.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚46073.13134765625 network.recv‚883359 disk.nvme0n1.in‚ 669.43359375 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.28955078125$ memory_percent‚98.67715835571289 proc.cpu.threads‚17# disk.dm-0.out‚46073.13134765625 disk.sda1.in‚0 disk.drbd1777.in‚0" disk.nvme0c0n1.in‚ 669.43359375 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2375.99609375 disk.sda3.out‚ 2375.99609375& disk.nvme0n1.out‚46073.13134765625 disk.dm-0.in‚ 669.43359375 disk.rbd2.out‚ 90.771484375* proc.memory.percent‚17.397260665893555% disk./.usageGB‚1080.9250755310059 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 216.734375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2850.3671875 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.177821115474572wÄTò‚(:… ãÔ©ÕñÓ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦qg:ó‚(:¯ ãÔæŸôÕ network.sent‚ 4.728711e+06 network.recv‚889055) disk./.usagePercent‚72.54179415159186 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚2396.875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚98.67713451385498% disk./.usageGB‚1080.9251556396484& disk.nvme0n1.out‚46172.33837890625 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 1.28955078125( proc.memory.availableMB‚ 216.73828125 cpu‚25.17707819789334 disk.rbd0.out‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2850.3671875 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.rbd2.out‚ 90.833984375 disk.rbd3.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 669.43359375* proc.memory.percent‚17.397260665893555 disk.dm-0.in‚ 669.43359375( disk.nvme0c0n1.out‚46172.33837890625 disk.sda.out‚2396.875 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚2396.875 disk.nvme0n1.in‚ 669.43359375# disk.dm-0.out‚46172.33837890625 disk.sda2.out‚0 disk.rbd1.in‚0d5âô‚(:… ãÔœá›Ó gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8046 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{–3±=õ‚(:² ãÔ€îûÕ) disk./.usagePercent‚72.54179159151651% disk./.usageGB‚1080.9251174926758 disk.rbd0.in‚0* proc.memory.percent‚17.397260665893555 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 2397.5625 cpu‚25.176426717466487 network.recv‚893677# memory_percent‚98.6771821975708 network.sent‚ 4.734905e+06 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 669.43359375 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 2397.5625 disk.rbd1.out‚ 1.28955078125 disk.dm-1.out‚ 2397.5625 disk.nvme0n1.in‚ 669.43359375& disk.nvme0n1.out‚46173.06298828125 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875( disk.nvme0c0n1.out‚46173.06298828125 disk.dm-0.in‚ 669.43359375 disk.sda.in‚ 0.07421875( proc.memory.availableMB‚ 216.73046875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 90.845703125 disk.sda2.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚46173.06298828125" proc.memory.rssMB‚ 2850.3671875ùF݈2ö- _wandbruntime‚2528  _runtime‚2528TÞƒÉ÷à ' lr-AdamW/pg2‚0.00021702827918447954 trainer/global_step‚2349 % _timestamp‚1.7883974650307024e+09 _runtime‚2528.590562031 _step‚187 ' lr-AdamW/pg1‚0.00021702827918447954[TãzÎøÈ ' lr-AdamW/pg2‚0.00021702827918447954 trainer/global_step‚2349 % _timestamp‚1.7883974650307024e+09 _runtime‚2528.590562031 _step‚187 ' lr-AdamW/pg1‚0.00021702827918447954»êP32ù- _wandbruntime‚2528  _runtime‚2528r³Ú§Ð'úÊO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43698273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25624058 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4322961392128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61949531 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874204672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4272971098112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25624060 % _timestamp‚1.7883974650317361e+09 _runtime‚2528.592004039 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88991499744768 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4272971098112 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42222294 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 287380480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105647802 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43698273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93264237258240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87952102972967 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4272737513472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42223022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚188 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16598236 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43698271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88991517747712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108044632385024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103721670992896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61975711 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88991499744768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4272737513472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105673984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103721647052800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87952348982171 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61949531 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93264488845824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88991517747712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83692247000064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4322697951744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83692229960704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4259873012263 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16598962 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108044345004544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 263440384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4260101982107 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16598236 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43698271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504841 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88991517747712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108044632385024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103721670992896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61975711 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88991499744768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4272737513472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105673984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32031 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103721647052800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87952348982171 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61949531 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93264488845824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88991517747712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83692247000064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4322697951744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83692229960704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4259873012263 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16598962 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108044345004544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 263440384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4260101982107 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32031 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4272971098112 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42222294 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 287380480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105647802 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43698273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93264237258240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87952102972967 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4272737513472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42223022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚188 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105718264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43716452 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93303440297472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.smal¾ûü"l_pool.allocated‚16606016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4261676745703 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61975606 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83727156654080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4261905715755 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62001812 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105692056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43716450 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4274780259840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43716452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43716450 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87988833399783 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105718264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108089837380096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4274546661888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89028642034688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108089552110080 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89028642034688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4274546661888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚2529.616158063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89028660037632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83727173693440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _step‚189 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2349 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62001812 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87989079409195 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4274780259840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25634713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16605327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42240038 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93303440297472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89028660037632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103765049902592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105692056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25634711 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43716452 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93303440297472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16606016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4261676745703 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61975606 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009412 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83727156654080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4261905715755 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62001812 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105692056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43716450 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4274780259840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43716452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43716450 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87988833399783 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26206 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105718264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108089837380096 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4274546661888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89028642034688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108089552110080 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89028642034688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4274546661888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚2529.616158063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89028660037632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83727173693440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _step‚189 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2349 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62001812 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87989079409195 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4274780259840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25634713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16605327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285270016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42240038 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93303440297472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89028660037632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103765049902592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105692056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25634711 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105718264 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1½$´2ÿ- _wandbruntime‚2529  _runtime‚2529jÌX’T€Î .loss/FBt3/detection_loss‚3.410965234041214 _runtime‚ 2529.61751489 &loss/total_loss‚60.179588001086664 /loss/FRt2/assignment_loss‚4.360627851178569 /loss/FRt3/assignment_loss‚5.799761056900024 /loss/FRt3/detection_loss‚2.5812097838834713 0loss/FBt1/assignment_loss‚0.4352954784892948 1loss/SRqqt2/detection_loss‚1.4536373168230057 trainer/global_step‚2349 0loss/SRqqt2/assignment_loss‚2.43864622555281 1loss/FBt4/assignment_loss‚0.29764368459784235 .loss/FRt1/detection_loss‚2.752638037715639 _step‚190 $ _timestamp‚1.788397466057665e+09 0loss/SRqqt1/detection_loss‚2.743115260487511 1loss/SRqqt3/detection_loss‚2.7219146611513914 /loss/FBt1/detection_loss‚1.9676265231136938 /loss/FBt4/detection_loss‚1.2069158577919006 /loss/FRt2/detection_loss‚2.0806397415937914 -loss/FRt1/assignment_loss‚5.9828827721732 .loss/FRt4/detection_loss‚2.251447859737608 1loss/FBt2/assignment_loss‚0.12311108410358429 epoch‚1 0loss/FRt4/assignment_loss‚2.9935380617777505 0loss/FBt3/assignment_loss‚0.6892436798661947 2loss/SRqqt1/assignment_loss‚3.1157108658836004 2loss/SRqqt3/assignment_loss‚2.7172314723332724 2loss/SRqqt4/assignment_loss‚2.3494010362131843 1loss/SRqqt4/detection_loss‚1.1328913326236023 .loss/FBt2/detection_loss‚4.573493123054504ûHYYÓ .loss/FBt3/detection_loss‚3.410965234041214 _runtime‚ 2529.61751489 &loss/total_loss‚60.179588001086664 /loss/FRt2/assignment_loss‚4.360627851178569 /loss/FRt3/assignment_loss‚5.799761056900024 /loss/FRt3/detection_loss‚2.5812097838834713 0loss/FBt1/assignment_loss‚0.4352954784892948 1loss/SRqqt2/detection_loss‚1.4536373168230057 trainer/global_step‚2349 0loss/SRqqt2/assignment_loss‚2.43864622555281 1loss/FBt4/assignment_loss‚0.29764368459784235 .loss/FRt1/detection_loss‚2.752638037715639 _step‚190 $ _timestamp‚1.788397466057665e+09 0loss/SRqqt1/detection_loss‚2.743115260487511 1loss/SRqqt3/detection_loss‚2.7219146611513914 /loss/FBt1/detection_loss‚1.9676265231136938 /loss/FBt4/detection_loss‚1.2069158577919006 /loss/FRt2/detection_loss‚2.0806397415937914 -loss/FRt1/assignment_loss‚5.9828827721732 .loss/FRt4/detection_loss‚2.251447859737608 1loss/FBt2/assignment_loss‚0.12311108410358429 epoch‚1 0loss/FRt4/assignment_loss‚2.9935380617777505 0loss/FBt3/assignment_loss‚0.6892436798661947 2loss/SRqqt1/assignment_loss‚3.1157108658836004 2loss/SRqqt3/assignment_loss‚2.7172314723332724 2loss/SRqqt4/assignment_loss‚2.3494010362131843 1loss/SRqqt4/detection_loss‚1.1328913326236023 .loss/FBt2/detection_loss‚4.573493123054504¾Á¿…c›‚‚(: ŸãÔ‘ˆýÔ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.071199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u <0Lƒ‚(:Á ŸãÔÑ’Ö disk.sda.out‚ 2412.59765625 proc.cpu.threads‚17 cpu‚25.175683037134046( disk.nvme0c0n1.out‚46173.50146484375# disk.dm-0.out‚46173.50146484375 network.sent‚ 4.740043e+06% disk./.usageGB‚1080.9255905151367 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚72.54182333645075 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 669.43359375* proc.memory.percent‚17.397260665893555 disk.sda2.out‚0 disk.rbd2.out‚90.93212890625( proc.memory.availableMB‚ 216.69921875 disk.sda3.in‚ 0.07421875 disk.nvme0n1.in‚ 669.43359375& disk.nvme0n1.out‚46173.50146484375 disk.sda.in‚ 0.07421875$ memory_percent‚98.67737293243408 disk.sda3.out‚ 2412.59765625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 1.28955078125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 669.43359375 disk.dm-1.out‚ 2412.59765625" proc.memory.rssMB‚ 2850.3671875 network.recv‚897247 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0ŸŽ„‚(:ƒ ®ãÔýž Ó gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚75.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ºg™;…‚(:° ®ãÔ‹ÛñÕ network.sent‚ 4.84827e+06 disk.rbd3.out‚0( disk.nvme0c0n1.out‚46272.89404296875 disk.sda.out‚2430.875' proc.memory.availableMB‚ 216.4453125% disk./.usageGB‚1080.9252471923828" disk.nvme0c0n1.in‚ 669.43359375 disk.rbd0.in‚0# disk.dm-0.out‚46272.89404296875 disk.drbd1777.out‚0) disk./.usagePercent‚72.54180029577267 disk.rbd1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.397260665893555 disk.nvme0n1.in‚ 669.43359375 disk.dm-0.in‚ 669.43359375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚90.94775390625 disk.sda3.out‚2430.875 disk.dm-1.in‚ 0.07421875& disk.nvme0n1.out‚46272.89404296875 disk.sda.in‚ 0.07421875 network.recv‚903477 disk.dm-1.out‚2430.875" proc.memory.rssMB‚ 2850.3671875 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.out‚0 disk.drbd1777.in‚0 cpu‚25.175053205946995 disk.sda2.out‚0 disk.rbd1.out‚ 1.28955078125$ memory_percent‚98.67892265319824Ï×\†‚(:„ ½ãÔÌÚæÒ gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚90.365' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.073 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©B–oE‡‚(:º ½ãÔ¥±ÆÖ$ disk.nvme0n1.out‚46273.294921875 disk.rbd2.in‚0& disk.nvme0c0n1.out‚46273.294921875 disk.rbd2.out‚91.07275390625# memory_percent‚98.6774206161499! disk.dm-0.out‚46273.294921875 disk.sda.out‚ 2448.20703125 network.sent‚ 4.853852e+06% disk./.usageGB‚1080.9251670837402 disk.sda2.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 669.43359375 disk.rbd3.out‚0 network.recv‚907652 disk.sda3.out‚ 2448.20703125 disk.rbd1.out‚ 1.28955078125" disk.nvme0c0n1.in‚ 669.43359375 disk.sda1.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 216.69140625 cpu‚25.174316511171227) disk./.usagePercent‚72.54179491961446 disk.drbd1777.in‚0* proc.memory.percent‚17.397260665893555 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2850.3671875 disk.sda.in‚ 0.07421875 disk.sda1.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 2448.20703125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 669.43359375®q§¶ˆ‚(:… ÌãÔõþûÔ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚81.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.3238 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê•ÌýI‰‚(:¾ ÌãÔçëÕ disk.rbd2.out‚92.64697265625" disk.nvme0c0n1.in‚ 669.43359375( disk.nvme0c0n1.out‚46274.23486328125 disk.dm-1.out‚ 2449.67578125# disk.dm-0.out‚46274.23486328125# memory_percent‚98.6774206161499* proc.memory.percent‚17.397260665893555$ disk./.usageGB‚1080.925148010254 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 669.43359375 disk.sda.in‚ 0.07421875" proc.memory.rssMB‚ 2850.3671875 cpu‚25.17361481616787 disk.sda1.out‚0 disk.sda3.out‚ 2449.67578125& disk.nvme0n1.out‚46274.23486328125 disk.sda2.out‚0 disk.rbd1.out‚ 1.28955078125 disk.rbd0.in‚0 disk.dm-0.in‚ 669.43359375( proc.memory.availableMB‚ 216.69140625 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚912438 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 disk.sda.out‚ 2449.67578125 proc.cpu.threads‚17 network.sent‚ 4.860242e+06) disk./.usagePercent‚72.54179363957678 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875äÁ¼µ2Š- _wandbruntime‚2582  _runtime‚2582 ؾÇ‹Á & lr-AdamW/pg1‚0.0002164969837437222 & lr-AdamW/pg2‚0.0002164969837437222 trainer/global_step‚2399 % _timestamp‚1.7883975186624534e+09 _runtime‚2582.222312111 _step‚191{RþÌŒÆ trainer/global_step‚2399 % _timestamp‚1.7883975186624534e+09 _runtime‚2582.222312111 _step‚191 & lr-AdamW/pg1‚0.0002164969837437222 & lr-AdamW/pg2‚0.0002164969837437222¿Ö¯2- _wandbruntime‚2582  _runtime‚2582ñˆšjtŽÊO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44607221 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44607223 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63282813 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90848632243712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4363442932736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16943666 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4414124194816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90848614240768 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105890616477696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚89788637904975 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4363209348096 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95212075176448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44607223 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63256633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283186176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4350302245315 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 _runtime‚2582.223554894 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107863854 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43099638 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259246080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_íhà\train_batch_start/segment.all.peak‚560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 trainer/global_step‚2399 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107890036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63256633 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110304716732416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4350073275471 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110304999918592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95211823588864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90848632243712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4363442932736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85438564629504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89788883914179 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4363209348096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43100376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26156708 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105890592537600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95211823588864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4414383440896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 % _timestamp‚1.7883975186633742e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90848614240768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584a2ƒÜÕ'ÏO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354433 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4363209348096 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95212075176448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31806 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44607223 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63256633 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26182 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283186176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4350302245315 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 _runtime‚2582.223554894 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107863854 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43099638 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259246080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 trainer/global_step‚2399 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107890036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63256633 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26182 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110304716732416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4350073275471 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31806 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233584640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110304999918592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95211823588864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251587584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90848632243712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4363442932736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233584640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85438564629504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89788883914179 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4363209348096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43100376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26156708 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105890592537600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95211823588864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4414383440896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009204 % _timestamp‚1.7883975186633742e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90848614240768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251587584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44607221 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44607223 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63282813 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90848632243712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4363442932736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16943666 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4414124194816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90848614240768 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874204672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105890616477696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚89788637904975 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560À0#E[2- _wandbruntime‚2583  _runtime‚2583¥ wêÚ&‘ÔM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2399 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107934316 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95250775027200 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95251026628096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16949620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44625400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4416214081024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2583.256068317 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107908108 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105933971447296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90885774533632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4365252094464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90885756530688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4351877008911 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85473491322880 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4365018496512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 _step‚193 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90885756530688 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89825614341203 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4365252094464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63308914 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16950336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110349920004096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4365018496512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107934316 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43117699 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26167361 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44625402 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚89825368331791 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚716 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90885774533632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43116981 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105933995387392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0Îú™N ’ÙM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44625402 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚89825368331791 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251600896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚716 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90885774533632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43116981 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105933995387392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2399 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107934316 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95250775027200 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251600896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163200 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95251026628096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16949620 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32031 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44625400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289464320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4416214081024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2583.256068317 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874204672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107908108 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105933971447296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970052 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90885774533632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4365252094464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233597952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31806 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90885756530688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233597952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4351877008911 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265524224 XJDeviceStatsMonitor.on_train_batch_end/inactive_splçåÛ=? it_bytes.small_pool.peak‚ 295630336 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85473491322880 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874204672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4365018496512 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26206 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504841 _step‚193 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90885756530688 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89825614341203 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4365252094464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63308914 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16950336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110349920004096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4365018496512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107934316 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43117699 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26167361 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354433 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253ÁDTÜ2“- _wandbruntime‚2583  _runtime‚25834Ʋ[”Õ .loss/FBt3/detection_loss‚2.760802169640859 /loss/FBt4/detection_loss‚1.0623406922795635 epoch‚1 0loss/FBt1/assignment_loss‚0.5174817939254648 1loss/FBt4/assignment_loss‚0.26420642969214503 /loss/FRt4/detection_loss‚1.8926038989361296 1loss/SRqqt3/detection_loss‚2.7918741524219515 /loss/FRt1/assignment_loss‚6.103016058603923 /loss/FRt3/assignment_loss‚5.958891410094041 %loss/total_loss‚61.69364824250097 2loss/SRqqt2/assignment_loss‚2.4911790054578047 /loss/FRt3/detection_loss‚2.3022198103941403 1loss/SRqqt1/detection_loss‚3.5411275885999203 /loss/FRt4/assignment_loss‚2.348032422522281 0loss/SRqqt4/detection_loss‚1.263009304652013 _runtime‚2583.257980501 _step‚194 0loss/FRt2/assignment_loss‚4.5203574749888205 /loss/FBt1/detection_loss‚1.4538298060966504 trainer/global_step‚2399 % _timestamp‚1.7883975196978455e+09 2loss/SRqqt1/assignment_loss‚2.9042183309793472 2loss/SRqqt3/assignment_loss‚2.7489117880662284 2loss/SRqqt4/assignment_loss‚2.5668310418904547 1loss/FBt2/assignment_loss‚0.40838971734046936 0loss/FBt3/assignment_loss‚0.7156721890920986 /loss/FRt1/detection_loss‚2.6627521294134633 /loss/FRt2/detection_loss‚2.4330317110431436 1loss/SRqqt2/detection_loss‚1.5335299265690339 .loss/FBt2/detection_loss‚6.449339389801025^¦”O`•Ú /loss/FRt1/assignment_loss‚6.103016058603923 /loss/FRt3/assignment_loss‚5.958891410094041 %loss/total_loss‚61.69364824250097 2loss/SRqqt2/assignment_loss‚2.4911790054578047 /loss/FRt3/detection_loss‚2.3022198103941403 1loss/SRqqt1/detection_loss‚3.5411275885999203 /loss/FRt4/assignment_loss‚2.348032422522281 0loss/SRqqt4/detection_loss‚1.263009304652013 _runtime‚2583.257980501 _step‚194 0loss/FRt2/assignment_loss‚4.5203574749888205 /loss/FBt1/detection_loss‚1.4538298060966504 trainer/global_step‚2399 % _timestamp‚1.7883975196978455e+09 2loss/SRqqt1/assignment_loss‚2.9042183309793472 2loss/SRqqt3/assignment_loss‚2.7489117880662284 2loss/SRqqt4/assignment_loss‚2.5668310418904547 1loss/FBt2/assignment_loss‚0.40838971734046936 0loss/FBt3/assignment_loss‚0.7156721890920986 /loss/FRt1/detection_loss‚2.6627521294134633 /loss/FRt2/detection_loss‚2.4330317110431436 1loss/SRqqt2/detection_loss‚1.5335299265690339 .loss/FBt2/detection_loss‚6.449339389801025 .loss/FBt3/detection_loss‚2.760802169640859 /loss/FBt4/detection_loss‚1.0623406922795635 epoch‚1 0loss/FBt1/assignment_loss‚0.5174817939254648 1loss/FBt4/assignment_loss‚0.26420642969214503 /loss/FRt4/detection_loss‚1.8926038989361296 1loss/SRqqt3/detection_loss‚2.7918741524219515Âö¿GŒ–‚(:„ ÛãÔŸï¹Ô gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚90.365' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.073 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DæY©H—‚(:½ ÛãÔÚÀÕÕ disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 669.43359375 disk.drbd1777.out‚0 network.sent‚ 4.865217e+06 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 669.43359375* proc.memory.percent‚17.397260665893555 cpu‚25.172911021681646 proc.cpu.threads‚17$ disk./.usageGB‚1080.925148010254 disk.rbd2.out‚92.80322265625( disk.nvme0c0n1.out‚46373.47509765625 disk.sda.out‚ 2466.3515625( proc.memory.availableMB‚ 216.62890625 network.recv‚915780# disk.dm-0.out‚46373.47509765625 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2850.3671875 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 2466.3515625 disk.dm-1.out‚ 2466.3515625 disk.rbd0.in‚0& disk.nvme0n1.out‚46373.47509765625 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0) disk./.usagePercent‚72.54179363957678 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 669.43359375 disk.sda1.out‚0$ memory_percent‚98.67780208587646 disk.rbd1.out‚ 1.289550781253ó˜‚(:„ êãÔÏ«ÖÓ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚87.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fø†§K™‚(:À êãÔðÌÇÖ disk.rbd3.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚46374.15966796875 disk.sda.in‚ 0.07421875 network.recv‚922315 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚46374.15966796875 disk.sda1.in‚0* proc.memory.percent‚17.452478408813477 cpu‚25.183287974832535 disk.drbd1777.out‚0( proc.memory.availableMB‚ 207.36328125 proc.cpu.threads‚17 disk.sda3.out‚ 2505.87109375) disk./.usagePercent‚72.54185917750556 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 669.43359375 disk.dm-1.out‚ 2505.87109375 disk.sda1.out‚0 disk.rbd2.out‚92.81103515625" proc.memory.rssMB‚ 2859.4140625 network.sent‚ 4.972762e+06$ disk./.usageGB‚1080.926124572754 disk.sda2.in‚0 disk.rbd1.out‚ 1.28955078125 disk.nvme0n1.in‚ 669.43359375& disk.nvme0n1.out‚46374.15966796875 disk.sda.out‚ 2505.87109375 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 669.43359375$ memory_percent‚98.73435497283936Z¶Áõš‚(:„ ùãÔ«þÓ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚81.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.2458 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ÞëI›‚(:¾ ùãÔˆÑþÕ( disk.nvme0c0n1.out‚46473.26513671875& disk.nvme0n1.out‚46473.26513671875# disk.dm-0.out‚46473.26513671875 disk.sda.in‚ 0.07421875# memory_percent‚98.5954999923706* proc.memory.percent‚17.314672470092773 disk.drbd1777.out‚0 disk.rbd2.out‚ 92.9130859375 disk.nvme0n1.in‚ 669.43359375( proc.memory.availableMB‚ 230.11328125 disk.sda2.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚72.54178672737336% disk./.usageGB‚1080.9250450134277 disk.rbd3.out‚0 disk.rbd1.out‚ 1.28955078125 disk.sda1.out‚0" proc.memory.rssMB‚ 2836.8359375 network.sent‚ 4.978248e+06 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 2517.16796875" disk.nvme0c0n1.in‚ 669.43359375 disk.dm-1.in‚ 0.07421875 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 2517.16796875 disk.rbd0.in‚0 disk.dm-0.in‚ 669.43359375 disk.sda.out‚ 2517.16796875 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 cpu‚25.20781317875214 network.recv‚926259©«½œ‚(:„ ˆãÔψÇÓ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚87.948' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ÂB=‚(:² ˆãԀѵÖ) disk./.usagePercent‚72.54178595935076 disk.sda1.in‚0 disk.rbd1.out‚ 1.28955078125 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚98.59662055969238" proc.memory.rssMB‚ 2836.8359375 network.recv‚930493 disk.drbd1777.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 4.984553e+06 disk.dm-1.out‚ 2550.1875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 669.43359375 disk.rbd2.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.314672470092773& disk.nvme0n1.out‚46473.70166015625 disk.dm-0.in‚ 669.43359375 disk.sda1.out‚0 disk.rbd2.out‚ 92.9130859375 disk.rbd0.in‚0# disk.dm-0.out‚46473.70166015625 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 669.43359375( disk.nvme0c0n1.out‚46473.70166015625 disk.rbd1.in‚0 disk.sda.out‚ 2550.1875$ disk./.usageGB‚1080.925033569336 disk.dm-1.in‚ 0.07421875 disk.sda3.out‚ 2550.1875' proc.memory.availableMB‚ 229.9296875 cpu‚25.229793018517867ÂœL”›ž‚(: —ãÔ¨ºõÒ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.276400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¼ÌPEŸ‚(:º —ãÔäðÙÕ$ disk./.usageGB‚1080.925048828125 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 669.43359375 disk.dm-0.in‚ 669.43359375 disk.rbd2.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚46473.97900390625 disk.sda.out‚ 2566.37890625! proc.memory.rssMB‚ 2843.296875( disk./.usagePercent‚72.5417869833809 disk.rbd1.out‚ 1.28955078125 disk.rbd3.out‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚46473.97900390625 disk.dm-1.out‚ 2566.37890625 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚98.63309860229492" disk.nvme0c0n1.in‚ 669.43359375 disk.sda.in‚ 0.07421875 disk.sda3.out‚ 2566.37890625# disk.dm-0.out‚46473.97900390625 disk.sda1.in‚0 disk.sda3.in‚ 0.07421875 network.recv‚934285 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 94.6435546875 disk.sda2.in‚0* proc.memory.percent‚17.354106903076172 network.sent‚ 4.989702e+06 disk.rbd3.in‚0& proc.memory.availableMB‚ 223.953125 cpu‚25.25294730452948qdF{2 - _wandbruntime‚2661  _runtime‚2661Á$ч7*¡±T REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44183402 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚112555965558784 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚560 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3516 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 130437632 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 517504841 % _timestamp‚1.7883975974977112e+09 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0”ÇVl¿ ¢¶T J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32031 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚560 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44183402 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚112555965558784 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚560 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3516 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 130437632 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 517504841 % _timestamp‚1.7883975974977112e+09 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32031 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚45718622 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17381256 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188851280 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚560 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚112584373274624 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4466279350272 _step‚196 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4415072022016 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4414896275456 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171811920 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6505365504 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚390 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚64195908 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 319570432 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚87370979012608 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6189583360 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚26817917 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚64175325 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 530579456 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚44198783 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32031 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚45718622 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17381256 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188851280 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚560 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚112584373274624 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4466279350272 _step‚196 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4415072022016 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4414896275456 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171811920 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6505365504 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚390 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944ÄÎ0ÀÌ›¦‚(: ¦ãÔ‡úµÓ gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.237' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.647399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡kz;§‚(:° ¦ãÔæžæÕ network.sent‚ 4.995973e+06 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 669.43359375 disk.sda1.out‚0 disk.rbd2.out‚ 94.6435546875 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.69394302368164" proc.memory.rssMB‚ 2852.7734375 disk.sda3.out‚ 2582.1875( disk.nvme0c0n1.out‚46573.34130859375 cpu‚25.277311932527944 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 669.43359375# disk.dm-0.out‚46573.34130859375) proc.memory.percent‚17.41194725036621 disk.rbd1.out‚ 1.28955078125 disk.dm-1.out‚ 2582.1875& disk.nvme0n1.out‚46573.34130859375$ disk./.usageGB‚1080.925136566162 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda.out‚ 2582.1875 network.recv‚938786) disk./.usagePercent‚72.54179287155418 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 669.43359375 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.07421875& proc.memory.availableMB‚ 213.984375Pƒ 12¨- _wandbruntime‚2677  _runtime‚2677z°+pŠ©Ÿ¤ 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12514619529247284 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_4_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.12224917113780975 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.7026346921920776 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.30412086844444275 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10143677890300751 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.2884615361690521 jet/accuracy_6_of_6‚0 jet/accuracy_2_of_7‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0016099568456411362 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2592816948890686 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30622759461402893 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.14500759541988373 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.06549064069986343 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.06293203681707382 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961197972297668 ;"Purity/*FRt1SRqqt1FBt/event_purity‚0.038995031267404556 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 *jet/accuracy_1_of_2‚0.5306822657585144 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 particle/accuracy_5_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1147172823548317 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.19298246502876282 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2583759129047394 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.13252325356006622 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.2557423412799835 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.6342687010765076 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1005050539970398 +jet/accuracy_3_of_5‚0.04139145091176033 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1458333283662796 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2552359998226166 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.0625 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.14220580458641052 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.1666666716337204 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.12004504352807999 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3059077858924866 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.988182783126831 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06088665872812271 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.13218404352664948 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.13638997077941895 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.10009577870368958 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.17627264559268951 A&Purity/1FRt*SRqqt4FBt/event_ ([»œtproportion‚0.00012400794366840273 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.1462624967098236 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.035555556416511536 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 epoch‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2765151560306549 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.1752873659133911 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.06275720149278641 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.01491860207170248 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0864039957523346 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.09181784093379974 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_4‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.14192256331443787 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.24045565724372864 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1250000149011612 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07183907926082611 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.15648441016674042 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0005120327696204185 (particle/accuracy‚0.8143807649612427 jet/accuracy_5_of_6‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.028619293123483658 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12270723283290863 *jet/accuracy_1_of_5‚0.8255120515823364 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.01515151560306549 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.002711640205234289 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.021810628473758698 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.17587009072303772 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09541192650794983 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2582723796367645 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9809328317642212 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.05000000074505806 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.03546099364757538 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3531290590763092 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06666772067546844 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985614001750946 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1409730315208435 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 +particle/specificity‚0.9198622703552246 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9907438158988953 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12017757445573807 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.29605263471603394 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2063492089509964 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_7‚1 <"Purity/2FRt0SRqqt1FBt/event_purity‚0.0035714285913854837 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚2417 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.262471079826355 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9854991436004639 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2357826828956604 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3031249940395355 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.24452677369117737 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9895685315132141 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14057239890098572 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1353408247232437 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13363364338874817 particle/accuracy_3_of_5‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.11666667461395264 *jet/accuracy_1_of_6‚0.6730769276618958 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2678571343421936 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0416666679084301 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07970656454563141 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,jet/accuracy_3_of_4‚0.027328617870807648 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.11237315833568573 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.16551651060581207 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0013216618681326509 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16439898312091827 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018298771232366562 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2307981550693512 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.1184854581952095 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9900919198989868 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3366907238960266 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.051212944090366364 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.2404443323612213 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.12605787813663483 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (particle/f_score‚0.29093870520591736 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.63956218957901 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.25516146421432495 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2591908574104309 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.03417570888996124 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2522988021373749 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.29651162028312683 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11699681729078293 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.27129626274108887 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.13868145644664764 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.12962962687015533 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.11059267818927765 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.1314886063337326 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12243621051311493 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.048914022743701935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0013227513991296291 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9929097890853882 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.02794833295047283 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.1542116403579712 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 _runtime‚2677.829388869 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.1462624967098236 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 _step‚197 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.05000000074505806 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.031198685988783836 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9626436829566956 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2259259670972824 particle/accuracy_2_of_7‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0011337868636474013 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3325550854206085 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.13027794659137726 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9774546027183533 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.20000000298023224 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990129292011261 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.035714291036129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9791666865348816 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.006578998640179634 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9916161298751831 jet/accuracy_4_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0396825410425663 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0016212905757129192 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.18446573615074158 particle/accuracy_4_of_6‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.23333333432674408 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1296013593673706 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9729070067405701 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12564890086650848 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.25200894474983215 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,jet/accuracy_4_of_5‚0.004704301245510578 particle/accuracy_2_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5378787517547607 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 jet/accuracy_1_of_7‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0747794583439827 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2649352252483368 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09967407584190369 particle/accuracy_1_of_7‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.048245616257190704 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1527777761220932 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9408982992172241 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.07330642640590668 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.1358024626970291 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.15964150428771973 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9903005361557007 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 )loss/val_total_loss‚59.13800800245854 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.047969866544008255 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.11387448012828827 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.32852041721343994 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0019841270986944437 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.262420117855072 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_2_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.27000001072883606 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99269700050354 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2648197114467621 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9955908060073853 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951466917991638 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.1255960613489151 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.25593242049217224 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.13726778328418732 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1216149777173996 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12016234546899796 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.03417570888996124 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.09288499504327774 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_3_of_6‚0.09615384787321091 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12443076074123383 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0572916679084301 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.20370370149612427 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.06178852543234825 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.125 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.12390086054801941 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.037686537951231 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09481366723775864 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_5_of_7‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11493641883134842 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.07570766657590866 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.2577252686023712 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 jet/accuracy_6_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 particle/accuracy_2_of_5‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.10577341169118881 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2440890073776245 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.335349440574646 <"Purity/0FRt2SRqqt0FBt/event_purity‚0.0032652532681822777 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9959086179733276 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9883256554603577 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.11391584575176239 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_6_of_6‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 +jet/accuracy_2_of_4‚0.21210485696792603 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2430555373430252 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.357840895652771 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.17390893399715424 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.10991284996271133 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2822916507720947 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.2723214030265808 particle/accuracy_5_of_5‚1 6validation_average_jet_accuracy‚0.2844775915145874 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.00374779524281621 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3060804307460785 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2986558973789215 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.991539478302002 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.13975603878498077 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.29635390639305115 particle/accuracy_3_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0674162432551384 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 jet/accuracy_7_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2247304767370224 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985075831413269 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11350381374359131 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2857142984867096 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.14406418800354004 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.17586208879947662 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0416666679084301 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.13375666737556458 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.14080891013145447 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8128049373626709 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15352723002433777 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.26232078671455383 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6057919859886169 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2742559313774109 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.14184576272964478 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2084341198205948 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.553030252456665 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22271491587162018 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9879794120788574 $ _timestamp‚1.788397614267206e+09?faåV ª¤¤ ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9774546027183533 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.20000000298023224 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.990129292011261 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.035714291036129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9791666865348816 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.006578998640179634 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9916161298751831 jet/accuracy_4_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0396825410425663 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0016212905757129192 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.18446573615074158 particle/accuracy_4_of_6‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.23333333432674408 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1296013593673706 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9729070067405701 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12564890086650848 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.25200894474983215 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,jet/accuracy_4_of_5‚0.004704301245510578 particle/accuracy_2_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088üz4­ù665671646595 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5378787517547607 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 jet/accuracy_1_of_7‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0747794583439827 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2649352252483368 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09967407584190369 particle/accuracy_1_of_7‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.048245616257190704 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1527777761220932 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9408982992172241 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.07330642640590668 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.1358024626970291 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.15964150428771973 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9903005361557007 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 )loss/val_total_loss‚59.13800800245854 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.047969866544008255 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.11387448012828827 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.32852041721343994 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0019841270986944437 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.262420117855072 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_2_of_6‚1 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.27000001072883606 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99269700050354 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2648197114467621 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9955908060073853 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951466917991638 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.1255960613489151 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.25593242049217224 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.13726778328418732 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1216149777173996 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12016234546899796 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.03417570888996124 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.09288499504327774 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_3_of_6‚0.09615384787321091 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.12443076074123383 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0572916679084301 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.20370370149612427 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.06178852543234825 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.125 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.12390086054801941 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.037686537951231 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09481366723775864 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_5_of_7‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11493641883134842 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.07570766657590866 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.2577252686023712 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 jet/accuracy_6_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 particle/accuracy_2_of_5‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.10577341169118881 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.2440890073776245 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.335349440574646 <"Purity/0FRt2SRqqt0FBt/event_purity‚0.0032652532681822777 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9959086179733276 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9883256554603577 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.11391584575176239 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_6_of_6‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 +jet/accuracy_2_of_4‚0.21210485696792603 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2430555373430252 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.357840895652771 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.17390893399715424 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.10991284996271133 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2822916507720947 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.2723214030265808 particle/accuracy_5_of_5‚1 6validation_average_jet_accuracy‚0.2844775915145874 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.00374779524281621 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3060804307460785 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2986558973789215 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.991539478302002 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.13975603878498077 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.29635390639305115 particle/accuracy_3_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0674162432551384 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 jet/accuracy_7_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2247304767370224 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.985075831413269 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11350381374359131 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2857142984867096 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.14406418800354004 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.17586208879947662 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0416666679084301 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.13375666737556458 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.14080891013145447 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8128049373626709 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15352723002433777 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.26232078671455383 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6057919859886169 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2742559313774109 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.14184576272964478 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2084341198205948 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.553030252456665 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22271491587162018 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9879794120788574 $ _timestamp‚1.788397614267206e+09 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.12514619529247284 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_4_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.12224917113780975 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.7026346921920776 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.30412086844444275 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10143677890300751 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.2884615361690521 jet/accuracy_6_of_6‚0 jet/accuracy_2_of_7‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0016099568456411362 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2592816948890686 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.30622759461402893 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.14500759541988373 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.06549064069986343 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.06293203681707382 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961197972297668 ;"Purity/*FRt1SRqqt1FBt/event_purity‚0.038995031267404556 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 *jet/accuracy_1_of_2‚0.5306822657585144 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 particle/accuracy_5_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1147172823548317 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.19298246502876282 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2583759129047394 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.13252325356006622 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.2557423412799835 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.6342687010765076 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1005050539970398 +jet/accuracy_3_of_5‚0.04139145091176033 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1458333283662796 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2552359998226166 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.0625 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.14220580458641052 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.1666666716337204 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.12004504352807999 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3059077858924866 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.988182783126831 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06088665872812271 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.13218404352664948 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.13638997077941895 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.10009577870368958 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.17627264559268951 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.1462624967098236 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.035555556416511536 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 epoch‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2765151560306549 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.1752873659133911 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.06275720149278641 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.01491860207170248 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0864039957523346 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.09181784093379974 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1111111119389534 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_4‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.14192256331443787 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.24045565724372864 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1250000149011612 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07183907926082611 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.15648441016674042 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0005120327696204185 (particle/accuracy‚0.8143807649612427 jet/accuracy_5_of_6‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.028619293123483658 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12270723283290863 *jet/accuracy_1_of_5‚0.8255120515823364 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.01515151560306549 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.002711640205234289 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.021810628473758698 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.17587009072303772 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09541192650794983 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2582723796367645 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9809328317642212 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.05000000074505806 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.03546099364757538 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3531290590763092 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06666772067546844 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985614001750946 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1409730315208435 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 +particle/specificity‚0.9198622703552246 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9907438158988953 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12017757445573807 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.29605263471603394 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2063492089509964 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_4_of_7‚1 <"Purity/2FRt0SRqqt1FBt/event_purity‚0.0035714285913854837 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚2417 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.262471079826355 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9854991436004639 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2357826828956604 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3031249940395355 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.24452677369117737 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9895685315132141 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14057239890098572 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1353408247232437 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13363364338874817 particle/accuracy_3_of_5‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.11666667461395264 *jet/accuracy_1_of_6‚0.6730769276618958 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2678571343421936 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0416666679084301 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.07970656454563141 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,jet/accuracy_3_of_4‚0.027328617870807648 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.11237315833568573 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.16551651060581207 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0013216618681326509 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16439898312091827 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018298771232366562 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2307981550693512 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.1184854581952095 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9900919198989868 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3366907238960266 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.051212944090366364 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.2404443323612213 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.12605787813663483 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (particle/f_score‚0.29093870520591736 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.63956218957901 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.25516146421432495 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2591908574104309 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.03417570888996124 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2522988021373749 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.29651162028312683 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11699681729078293 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.27129626274108887 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.13868145644664764 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.12962962687015533 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.11059267818927765 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.1314886063337326 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12243621051311493 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.048914022743701935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0013227513991296291 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9929097890853882 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.02794833295047283 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.1542116403579712 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 _runtime‚2677.829388869 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.1462624967098236 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/3FRt2SRqqpàÓÜt0FBt/event_proportion‚0.0024801588151603937 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 _step‚197 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.05000000074505806 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.031198685988783836 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9626436829566956 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2259259670972824 particle/accuracy_2_of_7‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0011337868636474013 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3325550854206085 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.13027794659137726Å¿ì"›«‚(: µãÔÔ½¢Ó gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚98.075' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.615000000000002 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýœ'TJ¬‚(:¿ µãÔóøÇÖ cpu‚25.28682685003944 network.sent‚ 5.103232e+06 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚46744.17333984375 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 2621.64453125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 94.7216796875 disk.nvme0n1.in‚ 669.43359375# disk.dm-0.out‚46744.17333984375$ memory_percent‚97.89502620697021( proc.memory.availableMB‚ 344.87890625$ disk./.usageGB‚1081.092441558838 disk.rbd0.out‚0& disk.nvme0n1.out‚46744.17333984375# proc.memory.rssMB‚ 2859.83203125 disk.sda.in‚ 0.07421875* proc.memory.percent‚17.455029487609863 disk.rbd3.out‚0 disk.dm-1.out‚ 2621.64453125 network.recv‚944670) disk./.usagePercent‚72.55302084999089 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.dm-0.in‚ 669.43359375 disk.sda.out‚ 2621.64453125 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.28955078125" disk.nvme0c0n1.in‚ 669.43359375òù×›­‚(: ÄãÔÄ‘âÒ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.909' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.181800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åß{B®‚(:· ÄãÔâàÕÕ disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚46852.2451171875 network.recv‚950782 disk.drbd1777.in‚0 disk.rbd2.out‚ 94.7216796875 disk.rbd1.out‚ 1.28955078125 disk.rbd3.out‚0' disk.nvme0c0n1.out‚46852.2451171875 disk.nvme0n1.in‚ 669.43359375* proc.memory.percent‚17.443013191223145 cpu‚25.285540961297656 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2622.453125 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.sent‚ 5.226843e+06) disk./.usagePercent‚72.55298142483062$ disk./.usageGB‚1081.091854095459 disk.sda1.out‚0 disk.sda3.in‚ 0.07421875$ memory_percent‚97.88329601287842( proc.memory.availableMB‚ 346.80078125" disk.dm-0.out‚46852.2451171875 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 669.43359375 disk.sda3.out‚ 2622.453125 disk.rbd1.in‚0 disk.dm-1.out‚ 2622.453125 disk.sda1.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 669.43359375# proc.memory.rssMB‚ 2857.86328125lòg«2¯- _wandbruntime‚2714  _runtime‚2714ägžÆ°À % _timestamp‚1.7883976510806546e+09 _runtime‚ 2714.64057951 _step‚198 & lr-AdamW/pg1‚0.0002159438085523774 & lr-AdamW/pg2‚0.0002159438085523774 trainer/global_step‚2449§{ï˱Š_step‚198 & lr-AdamW/pg1‚0.0002159438085523774 & lr-AdamW/pg2‚0.0002159438085523774 trainer/global_step‚2449 % _timestamp‚1.7883976510806546e+09 _runtime‚ 2714.64057951Æ“Ú”M2²- _wandbruntime‚2714  _runtime‚2714íÇ…PÐ'³ÊO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88783591951360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4461908153569 _runtime‚2714.641639968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4475418135552 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 % _timestamp‚1.7883976510815337e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94393420136960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46481333 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44940232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚88783574912000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27267446 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98868838272512 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111573426 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65092095 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94393420136960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17672784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114354996443648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109827528095744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 263439872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27267448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚94393438139904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4475418135552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44939511 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4475651720704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98869089860608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114355283823616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17672065 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚93245729074777 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65092095 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46481331 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚2449 _step‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4475651720704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109827504155648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46481331 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65118276 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98868838272512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4527755727872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307ÔŽÑÁÕ'´ÏO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚2449 _step‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4475651720704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109827504155648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46481331 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65118276 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98868838272512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4527755727872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88783591951360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4461908153569 _runtime‚2714.641639968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4475418135552 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 % _timestamp‚1.7883976510815337e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94393420136960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46481333 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44940232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚88783574912000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27267446 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98868838272512 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111573426 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65092095 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94393420136960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17672784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114354996443648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109827528095744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 263439872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27267448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚94393438139904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4475418135552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44939511 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4475651720704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98869089860608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114355283823616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17672065 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚93245729074777 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65092095 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46481331 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976Ç[¢½Lµ‚(:… ÓãÔ‰—úÒ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚78.616' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚15.7232 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊöVJ¶‚(:¿ ÓãÔˆ­¥Ö disk.sda3.out‚ 2639.87109375 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2639.87109375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 669.43359375& disk.nvme0n1.out‚47015.33837890625 disk.dm-0.in‚ 669.43359375 proc.cpu.threads‚17 network.sent‚ 5.232112e+06) disk./.usagePercent‚72.55297809673267 disk.sda1.out‚0# disk.dm-0.out‚47015.33837890625$ memory_percent‚97.88570404052734 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 2639.87109375 network.recv‚954648% disk./.usageGB‚1081.0918045043945 disk.rbd1.out‚ 1.28955078125 disk.sda.in‚ 0.07421875 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 669.43359375( disk.nvme0c0n1.out‚47015.33837890625# proc.memory.rssMB‚ 2857.86328125* proc.memory.percent‚17.443013191223145 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd2.out‚94.88623046875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 cpu‚25.284331434143127% proc.memory.availableMB‚ 346.40625ßH_€2·- _wandbruntime‚2715  _runtime‚2715ÕéAqS¸ÔM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4477460882432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4477227283968 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111643889 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94430562426880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44957562 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4529317541888 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44956848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98907789710848 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46499510 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111643889 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65118170 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚93282459501801 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚93282213492385 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94430580429824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17679461 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289463808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65144377 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17678749 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 trainer/global_step‚2449 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98908041312256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111617680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88818518644736 % _timestamp‚1.7883976520902934e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65118170 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184Rã²Pß&¹ÙM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94430562426880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4477227283968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46499512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46499510 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚88818501605376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114400200607232 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114400490071040 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111617680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109870907005440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44956848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98907789710848 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46499510 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111643889 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65118170 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚93282459501801 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚93282213492385 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94430580429824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17679461 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289463808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65144377 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17678749 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 trainer/global_step‚2449 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98908041312256 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111617680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88818518644736 % _timestamp‚1.7883976520902934e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65118170 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4477460882432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4477227283968 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111643889 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94430562426880 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44957562 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4529317541888 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100816676191232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 trainer/global_step‚2499 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66399197 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96250552635904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4566123555328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90529909580800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4565889970176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66425378 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚95082017997577 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100816424603136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18014098 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚95082264006785 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113815661 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27800096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100816424603136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116615766828032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111996473580544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4552337386625 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113789478 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4565889970176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18013357 Biò ÈÒO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47390281 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 WJDeviceStatsMonitor.on_trai‘lGTÏ&n_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚741 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4619031904768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 trainer/global_step‚2499 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66399197 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96250552635904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4566123555328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90529909580800 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4565889970176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66425378 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚95082017997577 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100816424603136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18014098 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚95082264006785 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113815661 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27800096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100816424603136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116615766828032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111996473580544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4552337386625 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113789478 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4565889970176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18013357 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100816676191232 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96287694925824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18020091 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4621122150912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100855627642880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4567699118592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45830840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90564836274176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47408460 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100855376041472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95118994433809 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113833732 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116660685177344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47408460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113833732 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4554141120273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4567932717056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4553912150217 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113859941 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96287676922880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚718 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113859941 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 _runtime‚2768.462064046 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚716 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4567932717056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95118748424393 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66451479 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27810751 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4567699118592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0€3{+ã&ËÝM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113833732 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116660685177344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47408460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113833732 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4554141120273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4567932717056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4553912150217 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113859941 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96287676922880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚718 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113859941 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 _runtime‚2768.462064046 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚716 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4567932717056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95118748424393 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66451479 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27810751 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4567699118592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96287676922880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47408462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45831558 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18020807 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96287694925824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18020091 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4621122150912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100855627642880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4567699118592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45830840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90564836274176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47408460 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚100855376041472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95118994433809 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ÌË!ºø2Ì- _wandbruntime‚2768  _runtime‚27689“³QÍË /loss/FRt3/assignment_loss‚6.115540806452433 /loss/FRt1/detection_loss‚2.8190848657063077 1loss/SRqqt4/detection_loss‚0.9974825901233337 trainer/global_step‚2499 2loss/SRqqt2/assignment_loss‚2.2234445730360544 1loss/FBt2/assignment_loss‚0.25227832794189453 /loss/FBt3/assignment_loss‚0.822979656276099 epoch‚2 % _timestamp‚1.7883977049035506e+09 _runtime‚ 2768.46326544 1loss/SRqqt1/assignment_loss‚2.950336075746096 2loss/SRqqt4/assignment_loss‚2.4538481951775877 0loss/FBt1/assignment_loss‚0.4765045130700814 /loss/FRt3/detection_loss‚2.9163311324392756 /loss/SRqqt2/detection_loss‚1.53873029296718 .loss/FRt2/detection_loss‚2.110586163298837 /loss/FRt4/assignment_loss‚2.536888286471367 2loss/SRqqt3/assignment_loss‚2.9699707952412693 1loss/SRqqt1/detection_loss‚2.9881203220440793 _step‚205 .loss/FBt3/detection_loss‚3.283872816297743 -loss/FBt4/detection_loss‚1.42190999471479 %loss/total_loss‚60.43943055908025 /loss/FRt2/assignment_loss‚4.045330164761379 /loss/FRt4/detection_loss‚2.1948419927308955 .loss/FBt2/detection_loss‚4.456506371498108 /loss/FRt1/assignment_loss‚6.243617352985201 1loss/FBt4/assignment_loss‚0.21180562507508713 0loss/SRqqt3/detection_loss‚2.910720493305813 .loss/FBt1/detection_loss‚1.498699151719341Wß åVÎÐ /loss/FRt2/assignment_loss‚4.045330164761379 /loss/FRt4/detection_loss‚2.1948419927308955 .loss/FBt2/detection_loss‚4.456506371498108 /loss/FRt1/assignment_loss‚6.243617352985201 1loss/FBt4/assignment_loss‚0.21180562507508713 0loss/SRqqt3/detection_loss‚2.910720493305813 .loss/FBt1/detection_loss‚1.498699151719341 /loss/FRt3/assignment_loss‚6.115540806452433 /loss/FRt1/detection_loss‚2.8190848657063077 1loss/SRqqt4/detection_loss‚0.9974825901233337 trainer/global_step‚2499 2loss/SRqqt2/assignment_loss‚2.2234445730360544 1loss/FBt2/assignment_loss‚0.25227832794189453 /loss/FBt3/assignment_loss‚0.822979656276099 epoch‚2 % _timestamp‚1.7883977049035506e+09 _runtime‚ 2768.46326544 1loss/SRqqt1/assignment_loss‚2.950336075746096 2loss/SRqqt4/assignment_loss‚2.4538481951775877 0loss/FBt1/assignment_loss‚0.4765045130700814 /loss/FRt3/detection_loss‚2.9163311324392756 /loss/SRqqt2/detection_loss‚1.53873029296718 .loss/FRt2/detection_loss‚2.110586163298837 /loss/FRt4/assignment_loss‚2.536888286471367 2loss/SRqqt3/assignment_loss‚2.9699707952412693 1loss/SRqqt1/detection_loss‚2.9881203220440793 _step‚205 .loss/FBt3/detection_loss‚3.283872816297743 -loss/FBt4/detection_loss‚1.42190999471479 %loss/total_loss‚60.43943055908025Í7–_-Ï‚(:… ‘ãÔ—ŸÓ gpu.0.gpu‚38.0Sƒ…c gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×¥ÿ’AЂ(:¶ ‘ãÔ«­Ö proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 5.437899e+06 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 2712.546875' proc.memory.availableMB‚ 464.7109375 cpu‚25.279494450904664& disk.nvme0n1.out‚47216.27392578125 disk.drbd1777.in‚0 disk.rbd2.out‚ 96.640625) disk./.usagePercent‚72.55297937677034" disk.nvme0c0n1.in‚ 669.43359375 disk.nvme0n1.in‚ 669.43359375 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2857.86328125 network.recv‚975325 disk.rbd1.in‚0 disk.rbd1.out‚ 1.28955078125 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2712.546875 disk.sda.out‚ 2712.546875 disk.rbd3.in‚0$ memory_percent‚97.16362953186035% disk./.usageGB‚1081.0918235778809# disk.dm-0.out‚47216.27392578125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 669.43359375* proc.memory.percent‚17.443013191223145( disk.nvme0c0n1.out‚47216.27392578125 disk.rbd0.out‚0 disk.sda3.in‚ 0.07421875Ä^®EÑ‚(:… ž‘ãÔÖÓŒÓ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3434 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùà.TGÒ‚(:¼ ž‘ãÔ§ïìÖ proc.cpu.threads‚17 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2733.75390625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 669.43359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 2733.75390625 disk.rbd0.out‚0 disk.sda1.in‚0* proc.memory.percent‚17.443013191223145 disk.dm-0.in‚ 669.43359375 disk.sda.out‚ 2733.75390625 disk.rbd2.out‚ 96.640625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚47217.12255859375 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0) disk./.usagePercent‚72.55297374460459$ disk./.usageGB‚1081.091739654541 disk.rbd1.out‚ 1.28955078125& disk.nvme0n1.out‚47217.12255859375( proc.memory.availableMB‚ 372.45703125 network.recv‚982426 disk.rbd2.in‚0$ memory_percent‚97.72670269012451 network.sent‚ 5.546664e+06 disk.sda3.in‚ 0.07421875( disk.nvme0c0n1.out‚47217.12255859375 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2857.86328125" disk.nvme0c0n1.in‚ 669.43359375 disk.sda1.out‚0 cpu‚25.278258894399976é¤Q›Ó‚(: ­‘ãÔèàÜÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚79.729' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.945799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»Y—¯EÔ‚(:º ­‘ãÔ…‘ÐÕ disk.dm-1.out‚ 2734.4140625 disk.rbd0.out‚0$ memory_percent‚98.32961559295654 disk.nvme0n1.in‚ 669.43359375 disk.dm-0.in‚ 669.43359375 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 669.43359375 proc.cpu.threads‚17 network.sent‚ 5.55242e+06 disk.sda1.in‚0 disk.rbd2.out‚ 96.77734375 disk.rbd3.out‚0# proc.memory.rssMB‚ 2857.86328125 cpu‚25.27707235937605& disk.nvme0n1.out‚47217.50244140625 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 2734.4140625 disk.sda2.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚47217.50244140625( proc.memory.availableMB‚ 273.67578125* proc.memory.percent‚17.443013191223145 network.recv‚986858 disk.rbd0.in‚0 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875# disk.dm-0.out‚47217.50244140625 disk.sda.out‚ 2734.4140625 disk.sda1.out‚0 disk.rbd1.out‚ 1.28955078125) disk./.usagePercent‚72.55298756901144% disk./.usageGB‚1081.0919456481934°/t1Õ‚(:… ¼‘ãÔÛ¿ÝÒ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0748 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Gf‡FÖ‚(:» ¼‘ãÔÆÝ­Ö& disk.nvme0n1.out‚47316.90673828125) disk./.usagePercent‚72.55298680098883% disk./.usageGB‚1081.0919342041016 disk.sda1.in‚0 disk.rbd2.out‚ 98.3515625( disk.nvme0c0n1.out‚47316.90673828125 disk.rbd3.out‚0 disk.sda3.out‚ 2757.1953125 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2757.1953125 disk.drbd1777.out‚0# disk.dm-0.out‚47316.90673828125# proc.memory.rssMB‚ 2857.86328125* proc.memory.percent‚17.443013191223145 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 669.43359375 disk.rbd3.in‚0 disk.drbd1777.in‚0" disk.nvme0c0n1.in‚ 669.43359375 disk.sda.in‚ 0.07421875 network.sent‚ 5.558698e+06 disk.rbd2.in‚0 disk.dm-0.in‚ 669.43359375 disk.sda2.out‚0 network.recv‚991530 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 2757.1953125( proc.memory.availableMB‚ 181.83984375 cpu‚25.275950212182927 proc.cpu.threads‚17 disk.rbd1.out‚ 1.28955078125$ memory_percent‚98.89013767242432N,H«2×- _wandbruntime‚2821  _runtime‚2821Íãë¿ÉØà trainer/global_step‚2549 % _timestamp‚1.7883977576329606e+09 _runtime‚2821.192855946 _step‚206 ' lr-AdamW/pg1‚0.00021477228457713156 ' lr-AdamW/pg2‚0.00021477228457713156Fÿ[TÎÙÈ % _timestamp‚1.7883977576329606e+09 _runtime‚2821.192855946 _step‚206 ' lr-AdamW/pg1‚0.00021477228457713156 ' lr-AdamW/pg2‚0.00021477228457713156 trainer/global_step‚2549ÎÌØÕ2Ú- _wandbruntime‚2821  _runtime‚2821ÆÜiÓ'ÛÍO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48299231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98107667131904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46692249 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48299233 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48299233 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118875821982208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4710692393984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4656595389952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18358758 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4642308679985 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67732480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92276261288960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4710426856960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 trainer/global_step‚2549 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4656361804800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96918552929585 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102764262521856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67706299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚745 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114165419065344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96918798938793 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48299231 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46691504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116005530 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98107667131904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18359501 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67706299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98107649128960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28332746 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92276244249600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4642537649833 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4656361804800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚2821.193799414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4656595389952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102764010933760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28332748 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102764010933760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚743 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 $ _timestamp‚1.788397757633753e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118876111459328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92276244249600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4642537649833 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4656361804800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚2821.193799414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4656595389952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102764010933760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28332748 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102764010933760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚743 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 $ _timestamp‚1.788397757633753e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118876111459328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4656595389952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18358758 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4642308679985 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67732480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92276261288960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4710426856960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 trainer/global_step‚2549 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4656361804800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96918552929585 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102764262521856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67706299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚745 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114165419065344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96918798938793 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48299231 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46691504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116005530 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98107667131904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18359501 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67706299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98107649128960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28332746 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÏGÎù2Ý- _wandbruntime‚2822  _runtime‚2822‡œ‡ÞØM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46709184 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96955529365817 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4644341383481 % _timestamp‚1.7883977586726408e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48317410 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114208774034944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48317410 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98144809421824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257135104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96955283356401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48317412 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28343401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281075200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98144809421824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18365785 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118921313531392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 QBDeviceStatsMonitor.on_train_batch_en›WJWd/active_bytes.large_pool.peak‚ 6408446976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4658170953216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4658170953216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4712258421248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4644112413425 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67758581 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98144791418880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118921032456192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102803213973504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116075993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48317412 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 _runtime‚2822.233390903 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46709885 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67732374 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98144791418880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92311170942976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4658404551680 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98144791418880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118921032456192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102803213973504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116075993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48317412 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 _runtime‚2822.233390903 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46709885 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67732374 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98144791418880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92311170942976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4658404551680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46709184 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96955529365817 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4644341383481 % _timestamp‚1.7883977586726408e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48317410 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114208774034944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48317410 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98144809421824 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257135104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96955283356401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48317412 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28343401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560Г3õ2à- _wandbruntime‚2822  _runtime‚2822!i•Aá» 1loss/FBt1/assignment_loss‚0.45849361241501657 loss/FBt2/assignment_loss‚0 .loss/FBt3/detection_loss‚2.977958224036477 _step‚209 .loss/FRt3/detection_loss‚2.654716020716088 .loss/FBt2/detection_loss‚2.965234935283661 %loss/total_loss‚58.41583477491792 trainer/global_step‚2549 .loss/FRt3/assignment_loss‚6.10732330594744 1loss/SRqqt3/detection_loss‚2.9511048247416816 /loss/FBt4/detection_loss‚1.3618367854505777 epoch‚2 2loss/SRqqt1/assignment_loss‚3.1774566769599915 1loss/SRqqt3/assignment_loss‚2.756706858674685 1loss/SRqqt4/assignment_loss‚2.186561323807273 1loss/SRqqt2/detection_loss‚1.3812665476659198 2loss/SRqqt2/assignment_loss‚2.4975273778409135 0loss/FBt3/assignment_loss‚0.4874219434442983 .loss/FRt2/detection_loss‚2.328625160873982 $ _timestamp‚1.788397758674873e+09 /loss/FRt2/assignment_loss‚4.356895036864699 1loss/FBt4/assignment_loss‚0.21312667990424453 .loss/FRt1/detection_loss‚2.665052741765976 .loss/FRt4/detection_loss‚2.414432735637177 0loss/SRqqt1/detection_loss‚2.924114856123924 1loss/SRqqt4/detection_loss‚1.3189157208716367 .loss/FBt1/detection_loss‚1.408843059075503 _runtime‚2822.234588671 /loss/FRt1/assignment_loss‚5.946547317504883 0loss/FRt4/assignment_loss‚2.8756730293118675„H$½FâÀ 1loss/SRqqt4/detection_loss‚1.3189157208716367 .loss/FBt1/detection_loss‚1.408843059075503 _runtime‚2822.234588671 /loss/FRt1/assignment_loss‚5.946547317504883 0loss/FRt4/assignment_loss‚2.8756730293118675 1loss/FBt1/assignment_loss‚0.45849361241501657 loss/FBt2/assignment_loss‚0 .loss/FBt3/detection_loss‚2.977958224036477 _step‚209 .loss/FRt3/detection_loss‚2.654716020716088 .loss/FBt2/detection_loss‚2.965234935283661 %loss/total_loss‚58.41583477491792 trainer/global_step‚2549 .loss/FRt3/assignment_loss‚6.10732330594744 1loss/SRqqt3/detection_loss‚2.9511048247416816 /loss/FBt4/detection_loss‚1.3618367854505777 epoch‚2 2loss/SRqqt1/assignment_loss‚3.1774566769599915 1loss/SRqqt3/assignment_loss‚2.756706858674685 1loss/SRqqt4/assignment_loss‚2.186561323807273 1loss/SRqqt2/detection_loss‚1.3812665476659198 2loss/SRqqt2/assignment_loss‚2.4975273778409135 0loss/FBt3/assignment_loss‚0.4874219434442983 .loss/FRt2/detection_loss‚2.328625160873982 $ _timestamp‚1.788397758674873e+09 /loss/FRt2/assignment_loss‚4.356895036864699 1loss/FBt4/assignment_loss‚0.21312667990424453 .loss/FRt1/detection_loss‚2.665052741765976 .loss/FRt4/detection_loss‚2.414432735637177 0loss/SRqqt1/detection_loss‚2.924114856123924Ñ'X"ôšã‚(: Ë‘ãÔîÅŽÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚77.13' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.425999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Î6ÇEä‚(:º Ë‘ãÔ¬ôåÕ disk.rbd3.in‚0 disk.sda3.out‚ 2794.99609375 disk.rbd1.out‚ 1.28955078125$ memory_percent‚98.93250465393066' proc.memory.availableMB‚ 174.8984375 network.recv‚994747$ disk./.usageGB‚1081.091896057129 disk.rbd2.out‚ 98.4296875 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875& disk.nvme0n1.out‚47317.28955078125 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 669.43359375 proc.cpu.threads‚17 network.sent‚ 5.563751e+06 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2794.99609375 disk.dm-1.in‚ 0.07421875 cpu‚25.27477610895671( disk./.usagePercent‚72.5529842409135" disk.nvme0c0n1.in‚ 669.43359375# disk.dm-0.out‚47317.28955078125* proc.memory.percent‚17.443013191223145 disk.sda1.out‚0 disk.dm-0.in‚ 669.43359375 disk.drbd1777.out‚0( disk.nvme0c0n1.out‚47317.28955078125 disk.dm-1.out‚ 2794.99609375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2857.86328125ÌW 1å‚(:… Ú‘ãÔ”¤ôÒ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.133' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8266 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿð’.Dæ‚(:¹ Ú‘ãÔºéÍÖ disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚47416.3994140625 disk.dm-0.in‚ 669.43359375" disk.dm-0.out‚47416.3994140625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875 disk.sda2.in‚0 disk.sda3.out‚ 2797.0703125" disk.nvme0c0n1.in‚ 669.43359375 network.sent‚ 5.671493e+06 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0* proc.memory.percent‚17.443013191223145 disk.rbd1.out‚ 1.28955078125 disk.nvme0n1.in‚ 669.43359375 disk.drbd1777.in‚0 disk.rbd2.out‚ 98.4296875 disk.rbd3.in‚0 network.recv‚ 1.001137e+06 disk.sda.out‚ 2797.0703125 disk.dm-1.in‚ 0.07421875$ memory_percent‚98.93407821655273 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 2797.0703125& proc.memory.availableMB‚ 174.640625 cpu‚25.27358926225073 disk.sda1.in‚0# proc.memory.rssMB‚ 2857.86328125' disk.nvme0c0n1.out‚47416.3994140625) disk./.usagePercent‚72.55298040080048# disk./.usageGB‚1081.09183883667y*ðùç‚(:… é‘ãÔÜÓÿÒ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3434 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-‘{Gè‚(:¼ é‘ãÔÍÒŠÖ disk.sda3.out‚ 2798.26171875% disk.nvme0c0n1.out‚47416.85546875 disk.nvme0n1.in‚ 669.43359375 disk.sda1.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.443013191223145) disk./.usagePercent‚72.55298168083814% disk./.usageGB‚1081.0918579101562 disk.dm-1.in‚ 0.07421875# disk.nvme0n1.out‚47416.85546875 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.28955078125 disk.rbd0.out‚0 disk.dm-0.in‚ 669.43359375 disk.rbd2.in‚0& proc.memory.availableMB‚ 174.890625" disk.nvme0c0n1.in‚ 669.43359375$ memory_percent‚98.93255233764648 cpu‚25.272525237154383 network.sent‚ 5.67702e+06 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 2798.26171875 disk.rbd0.in‚0 disk.dm-0.out‚47416.85546875 disk.sda.out‚ 2798.26171875 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2857.86328125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.07421875 proc.cpu.threads‚17 disk.rbd2.out‚98.53173828125 network.recv‚ 1.005046e+06Žš˜F2é- _wandbruntime‚2874  _runtime‚2874,nYÅê¿ % _timestamp‚1.7883978112959778e+09 _runtime‚2874.855862152 _step‚210 % lr-AdamW/pg1‚0.000214154179937071 % lr-AdamW/pg2‚0.000214154179937071 trainer/global_step‚2599Lö%ÊëÄ % lr-AdamW/pg1‚0.000214154179937071 % lr-AdamW/pg2‚0.000214154179937071 trainer/global_step‚2599 % _timestamp‚1.7883978112959778e+09 _runtime‚2874.855862152 _step‚210Ò3™2ì- _wandbruntime‚2874  _runtime‚2874tüjÒÓ'íÍO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104711597264384 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 % _timestamp‚1.7883978112968605e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚99964781627904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4801947529728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18703735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104711848852480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4732737913041 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69013401 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99964763624960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 trainer/global_step‚2599 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99964781627904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4746833639424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4747067224576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94022595957760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49208183 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69013401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47569861 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4732508943193 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69039582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49208183 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4746833639424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98755333870801 _step‚211 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69039582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28865398 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18704463 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4747067224576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚730 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47569131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28865396 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121136569228288 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104711597264384 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104711848852480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49208181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121136288139776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116334364550144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99964763624960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281088512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49208181 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118247765 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116334340610048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98755087861593 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94022578918400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 2874.85715549 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118221582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104711597264384 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104711848852480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49208181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121136288139776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116334364550144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99964763624960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281088512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49208181 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118247765 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116334340610048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98755087861593 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94022578918400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 2874.85715549 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118221582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚99964781627904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4801947529728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18703735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104711848852480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4732737913041 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69013401 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99964763624960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 trainer/global_step‚2599 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99964781627904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4746833639424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4747067224576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94022595957760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49208183 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69013401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47569861 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4732508943193 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69039582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49208183 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4746833639424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98755333870801 _step‚211Ó¶ÊÁ2ï- _wandbruntime‚2875  _runtime‚2875íùføá&ðÛM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94057522651136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4734312676633 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4803768451072 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _step‚212 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49226360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118292045 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18710940 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269718016 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4748876386304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98792064297825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94057505611776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4734541646689 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 _runtime‚2875.859214399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121181781628928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47586989 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4748642787840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69065683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98791818288409 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4748876386304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49226362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47587705 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118292045 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18710940 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269718016 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4748876386304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98792064297825 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94057505611776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4734541646689 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 _runtime‚2875.859214399 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121181781628928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47586989 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4748642787840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69065683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98791818288409 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4748876386304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49226362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47587705 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100591868802809 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4822938176249 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95768913587200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48446901 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70346684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70320503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123396614784000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106659435183104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70320503 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4893594226176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95768930626560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7883978638333292e+09 _step‚215 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50117131 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 trainer/global_step‚2649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50117131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106659183595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70346684 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29398048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101821878120960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19049585 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106659435183104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106659183595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48447633 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101821896123904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4893328689152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0$ªøÜß'‚ÙO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100591868802809 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4822938176249 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95768913587200 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48446901 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70346684 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70320503 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123396614784000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106659435183104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70320503 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4893594226176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95768930626560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7883978638333292e+09 _step‚215 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50117131 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 trainer/global_step‚2649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50117131 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106659183595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70346684 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29398048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101821878120960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19049585 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106659435183104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106659183595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48447633 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101821896123904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4893328689152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120437634 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123396904261120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118503286094848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4822709206401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4837305474048 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976×HS0½2ƒ- _wandbruntime‚2928  _runtime‚2928 õ³„ÞM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118546688944640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29408699 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4839114622464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50135310 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4839348220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19055342 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120481888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295755264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100628353220417 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 _runtime‚2928.397461101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101859020410880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106698135033344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106698386634752 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50135312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50135310 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120508097 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50135312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚730 trainer/global_step‚2649 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271815168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4895150460416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120481888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48464041 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95803857319936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70372785 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 _step‚216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4839114622464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3071‡fé&…ãM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 _step‚216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4839114622464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118546688944640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29408699 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4839114622464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50135310 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4839348220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19055342 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120481888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295755264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100628353220417 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 _runtime‚2928.397461101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101859020410880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106698135033344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106698386634752 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50135312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50135310 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120508097 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50135312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚730 trainer/global_step‚2649 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 271815168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4895150460416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120481888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48464041 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95803857319936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70372785ØÔÊ›b2†- _wandbruntime‚2928  _runtime‚2928kÉV‡Ð .loss/FBt2/detection_loss‚3.337475210428238 .loss/FRt1/assignment_loss‚6.03238911098904 1loss/SRqqt4/detection_loss‚1.2938799116905633 _runtime‚2928.398658087 1loss/SRqqt1/detection_loss‚2.9716943899790444 % _timestamp‚1.7883978648389373e+09 .loss/FRt3/detection_loss‚2.529636373449313 1loss/SRqqt2/detection_loss‚1.2810627538330701 /loss/FBt4/detection_loss‚1.2234243577612298 /loss/FRt3/assignment_loss‚6.080246624193694 2loss/SRqqt1/assignment_loss‚3.1384864350159964 1loss/FBt4/assignment_loss‚0.24423732660583092 1loss/SRqqt3/detection_loss‚3.0046659730947933 /loss/FBt1/detection_loss‚2.0029488814029985 1loss/SRqqt3/assignment_loss‚2.603328911157755 %loss/total_loss‚58.43239655054727 _step‚217 2loss/SRqqt4/assignment_loss‚2.2358230909743866 /loss/FBt1/assignment_loss‚0.422380995402879 0loss/FBt2/assignment_loss‚0.8848480544984341 0loss/FBt3/assignment_loss‚0.4581793912816052 /loss/FBt3/detection_loss‚2.9099583625793457 epoch‚2 trainer/global_step‚2649 0loss/FRt4/assignment_loss‚2.4667926260403226 /loss/FRt1/detection_loss‚2.7877543149171053 /loss/FRt2/detection_loss‚2.0671056945776116 .loss/FRt4/detection_loss‚2.120532587170601 0loss/FRt2/assignment_loss‚3.9251010561811515 0loss/SRqqt2/assignment_loss‚2.41044411732226b:Œ[ˆÕ 1loss/SRqqt4/detection_loss‚1.2938799116905633 _runtime‚2928.398658087 1loss/SRqqt1/detection_loss‚2.9716943899790444 % _timestamp‚1.7883978648389373e+09 .loss/FRt3/detection_loss‚2.529636373449313 1loss/SRqqt2/detection_loss‚1.2810627538330701 /loss/FBt4/detection_loss‚1.2234243577612298 /loss/FRt3/assignment_loss‚6.080246624193694 2loss/SRqqt1/assignment_loss‚3.1384864350159964 1loss/FBt4/assignment_loss‚0.24423732660583092 1loss/SRqqt3/detection_loss‚3.0046659730947933 /loss/FBt1/detection_loss‚2.0029488814029985 1loss/SRqqt3/assignment_loss‚2.603328911157755 %loss/total_loss‚58.43239655054727 _step‚217 2loss/SRqqt4/assignment_loss‚2.2358230909743866 /loss/FBt1/assignment_loss‚0.422380995402879 0loss/FBt2/assignment_loss‚0.8848480544984341 0loss/FBt3/assignment_loss‚0.4581793912816052 /loss/FBt3/detection_loss‚2.9099583625793457 epoch‚2 trainer/global_step‚2649 0loss/FRt4/assignment_loss‚2.4667926260403226 /loss/FRt1/detection_loss‚2.7877543149171053 /loss/FRt2/detection_loss‚2.0671056945776116 .loss/FRt4/detection_loss‚2.120532587170601 0loss/FRt2/assignment_loss‚3.9251010561811515 0loss/SRqqt2/assignment_loss‚2.41044411732226 .loss/FBt2/detection_loss‚3.337475210428238 .loss/FRt1/assignment_loss‚6.03238911098904Ùìæî8›‰‚(: ´’ãÔ¦ðôÓ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚76.485' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.296999999999999 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z5VÖDŠ‚(:¹ ´’ãÔ²åÖ# disk.dm-0.out‚47674.16357421875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚2882.625* proc.memory.percent‚17.443013191223145 proc.cpu.threads‚17 disk.sda.out‚2882.625" disk.nvme0c0n1.in‚ 669.43359375 cpu‚25.26706911997424) disk./.usagePercent‚71.97180284888461& disk.nvme0n1.out‚47674.16357421875( proc.memory.availableMB‚ 175.68359375 disk.rbd1.in‚0 network.sent‚ 5.808843e+06 network.recv‚ 1.028178e+06 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0! disk.rbd2.out‚100.32470703125# proc.memory.rssMB‚ 2857.86328125 disk.dm-0.in‚ 669.43359375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1.37548828125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚47674.16357421875 disk.dm-1.in‚ 0.07421875$ memory_percent‚98.92771244049072% disk./.usageGB‚1072.4318733215332 disk.nvme0n1.in‚ 669.43359375 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚2882.625 disk.rbd0.out‚0çüÆ‹‚(:… Ã’ãÔ›ŠæÒ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û…ÓOŒ‚(:Ä Ã’ãÔëÆäÕ disk.dm-0.in‚ 669.43359375 disk.rbd2.in‚0 disk.dm-1.in‚ 0.07421875 disk.sda.in‚ 0.07421875 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0 disk.dm-1.out‚ 2887.6640625 disk.rbd0.out‚0 disk.drbd1777.out‚0$ memory_percent‚98.92923831939697( proc.memory.availableMB‚ 175.43359375) disk./.usagePercent‚71.97180515295241 disk.rbd3.in‚0 disk.rbd2.out‚100.4892578125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 669.43359375 disk.rbd3.out‚0 disk.rbd1.out‚ 1.43798828125# disk.dm-0.out‚47674.48779296875 disk.nvme0n1.in‚ 669.43359375& disk.nvme0n1.out‚47674.48779296875 disk.drbd1777.in‚0( disk.nvme0c0n1.out‚47674.48779296875 network.recv‚ 1.03402e+06 disk.sda2.out‚0* proc.memory.percent‚17.443013191223145 cpu‚25.265981253576236 network.sent‚ 5.916272e+06 disk.sda3.out‚ 2887.6640625 disk.sda.out‚ 2887.6640625 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2857.86328125% disk./.usageGB‚1072.4319076538086 disk.sda2.in‚0 disk.rbd0.in‚0Ľfˆ‚(:… Ò’ãÔ¡®±Ô gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.112' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4224 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&+¢JŽ‚(:¿ Ò’ãÔòœ¢Ö disk.sda.out‚ 2888.3828125 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875( proc.memory.availableMB‚ 175.42578125 disk.dm-1.out‚ 2888.3828125# proc.memory.rssMB‚ 2857.86328125) disk./.usagePercent‚71.97181385720857" disk.nvme0c0n1.in‚ 669.43359375 disk.sda1.out‚0 disk.rbd3.in‚0% disk./.usageGB‚1072.4320373535156' disk.nvme0c0n1.out‚47773.9013671875 disk.dm-0.in‚ 669.43359375 network.recv‚ 1.03851e+06 disk.rbd0.out‚0% disk.nvme0n1.out‚47773.9013671875 disk.rbd2.in‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0" disk.dm-0.out‚47773.9013671875 disk.drbd1777.out‚0 disk.rbd2.out‚100.4892578125 disk.sda2.out‚0 disk.sda3.out‚ 2888.3828125 disk.rbd1.out‚ 1.43798828125 disk.nvme0n1.in‚ 669.43359375 disk.rbd1.in‚0# memory_percent‚98.9292860031128 cpu‚25.264975266316675 network.sent‚ 5.92258e+06 disk.sda.in‚ 0.07421875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.443013191223145 proc.cpu.threads‚17Ø6ýä2- _wandbruntime‚2980  _runtime‚2980P´—3Éà _step‚218 ' lr-AdamW/pg1‚0.00021285392938563309 ' lr-AdamW/pg2‚0.00021285392938563309 trainer/global_step‚2699 % _timestamp‚1.7883979165544791e+09 _runtime‚2980.114290374´ü cÎ‘È _step‚218 ' lr-AdamW/pg1‚0.00021285392938563309 ' lr-AdamW/pg2‚0.00021285392938563309 trainer/global_step‚2699 % _timestamp‚1.7883979165544791e+09 _runtime‚2980.114290374Ú'2’- _wandbruntime‚2980  _runtime‚2980R% jÚ'“ÔO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97515248256000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51026081 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4927777308672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122679869 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103679010619904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19394275 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120672231579648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97515265295360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49324274 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108607021513728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928010893824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51026083 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71627605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71653786 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7883979165552707e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71653786 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103678992616960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 250856960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102428403734817 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108607021513728 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125656972488192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102428157725609 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49324973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4984991765504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2980.115313363 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120672255519744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108606769925632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103678992616960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71627605 _step‚219 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103679010619904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19394275 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120672231579648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97515265295360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579jHÝÀ$456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49324274 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108607021513728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928010893824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51026083 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71627605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71653786 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7883979165552707e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71653786 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103678992616960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 250856960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102428403734817 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108607021513728 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125656972488192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102428157725609 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49324973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4984991765504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2980.115313363 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120672255519744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108606769925632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103678992616960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71627605 _step‚219 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122724149 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4986564058112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚735 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19401004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4914713203049 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51044260 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71679887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125702174547456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49342355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289463808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51044262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103716134906880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19400271 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71653680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2981.090141822 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4929586457088 trainer/global_step‚2699 % _timestamp‚1.7883979175295846e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122697940 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102465134161841 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120715634429440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29941351 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108645721363968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71653680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108645972965376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103716152909824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51044260 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120715610489344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108645972965376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4929586457088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29941349 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚220 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103716134906880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122697940 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49341620 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125702464011264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51044262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103716134906880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19400271 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71653680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2981.090141822 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4929586457088 trainer/global_step‚2699 % _timestamp‚1.7883979175295846e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122697940 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102465134161841 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120715634429440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29941351 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108645721363968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71653680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108645972965376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103716152909824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51044260 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120715610489344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108645972965376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4929586457088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29941349 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚220 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103716134906880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122697940 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49341620 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125702464011264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122724149 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4986564058112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚735 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19401004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4914713203049 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51044260 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71679887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125702174547456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49342355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289463808ÜŽ Ht2˜- _wandbruntime‚2981  _runtime‚29810Q]A™» /loss/FRt2/assignment_loss‚3.616879226984801 1loss/SRqqt2/assignment_loss‚2.121575785384459 /loss/FRt1/detection_loss‚2.7948531905810037 .loss/FRt4/detection_loss‚2.284003326550443 1loss/SRqqt2/detection_loss‚1.3302072996602339 0loss/SRqqt3/detection_loss‚3.133331537246704 0loss/SRqqt4/detection_loss‚1.329159947464595 /loss/FBt1/detection_loss‚1.3482178152676223 &loss/total_loss‚60.620296248490064 _runtime‚2981.091335713 /loss/FRt4/assignment_loss‚2.916814350067301 1loss/SRqqt4/assignment_loss‚2.216295829697235 loss/FBt2/assignment_loss‚0 $ _timestamp‚1.788397917531622e+09 2loss/SRqqt1/assignment_loss‚2.8520826250314713 0loss/FBt1/assignment_loss‚0.6088859926909562 1loss/SRqqt1/detection_loss‚2.8400040082633495 0loss/FBt4/assignment_loss‚0.3139411931173061 /loss/FBt4/detection_loss‚1.4761051973977755 epoch‚2 .loss/FRt1/assignment_loss‚6.31767695038407 /loss/FRt3/assignment_loss‚6.450172478502447 2loss/SRqqt3/assignment_loss‚3.0553873082002005 0loss/FBt3/assignment_loss‚0.3517283879693423 .loss/FRt2/detection_loss‚2.133137689696418 /loss/FRt3/detection_loss‚2.6904074868017975 .loss/FBt2/detection_loss‚4.747474312782288 _step‚221 .loss/FBt3/detection_loss‚3.691954308748245 trainer/global_step‚2699ðÇX™FšÀ .loss/FRt1/assignment_loss‚6.31767695038407 /loss/FRt3/assignment_loss‚6.450172478502447 2loss/SRqqt3/assignment_loss‚3.0553873082002005 0loss/FBt3/assignment_loss‚0.3517283879693423 .loss/FRt2/detection_loss‚2.133137689696418 /loss/FRt3/detection_loss‚2.6904074868017975 .loss/FBt2/detection_loss‚4.747474312782288 _step‚221 .loss/FBt3/detection_loss‚3.691954308748245 trainer/global_step‚2699 /loss/FRt2/assignment_loss‚3.616879226984801 1loss/SRqqt2/assignment_loss‚2.121575785384459 /loss/FRt1/detection_loss‚2.7948531905810037 .loss/FRt4/detection_loss‚2.284003326550443 1loss/SRqqt2/detection_loss‚1.3302072996602339 0loss/SRqqt3/detection_loss‚3.133331537246704 0loss/SRqqt4/detection_loss‚1.329159947464595 /loss/FBt1/detection_loss‚1.3482178152676223 &loss/total_loss‚60.620296248490064 _runtime‚2981.091335713 /loss/FRt4/assignment_loss‚2.916814350067301 1loss/SRqqt4/assignment_loss‚2.216295829697235 loss/FBt2/assignment_loss‚0 $ _timestamp‚1.788397917531622e+09 2loss/SRqqt1/assignment_loss‚2.8520826250314713 0loss/FBt1/assignment_loss‚0.6088859926909562 1loss/SRqqt1/detection_loss‚2.8400040082633495 0loss/FBt4/assignment_loss‚0.3139411931173061 /loss/FBt4/detection_loss‚1.4761051973977755 epoch‚2ÝHÖ>S›‚(:… á’ãÔ½»ïÒ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚99.253' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8506 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Šê¹œ‚(:À á’ãÔëÆÞÕ disk.rbd0.in‚0 disk.sda.out‚ 2888.91796875 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 175.3984375# pr†¬ j’oc.memory.rssMB‚ 2857.86328125 proc.cpu.threads‚17 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 669.43359375! disk.dm-0.out‚47774.263671875 disk.sda1.in‚0 network.sent‚ 5.928053e+06 network.recv‚ 1.042513e+06 disk.rbd1.in‚0 cpu‚25.263996555514034) disk./.usagePercent‚71.97181897735928 disk.sda1.out‚0 disk.sda3.out‚ 2888.91796875 disk.dm-1.out‚ 2888.91796875 disk.rbd0.out‚0$ disk.nvme0n1.out‚47774.263671875 disk.sda.in‚ 0.07421875 disk.rbd2.out‚102.2041015625 disk.rbd3.out‚0$ memory_percent‚98.92945289611816* proc.memory.percent‚17.443013191223145$ disk./.usageGB‚1072.432113647461" disk.nvme0c0n1.in‚ 669.43359375& disk.nvme0c0n1.out‚47774.263671875 disk.dm-0.in‚ 669.43359375 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.47021484375 disk.drbd1777.out‚0ü©ëK›‚(: ð’ãÔ²ÖûÒ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚75.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.191200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ÿ1Ož‚(:Ä ð’ãÔ¬šôÕ disk.sda2.in‚0 disk.dm-1.out‚ 2889.67578125" disk.dm-0.out‚47873.7197265625 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2889.67578125 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.47021484375# proc.memory.rssMB‚ 2857.86328125 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚47873.7197265625' proc.memory.availableMB‚ 175.3515625 disk.rbd2.out‚102.2041015625) disk./.usagePercent‚71.97181385720857 disk.dm-0.in‚ 669.43359375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 669.43359375* proc.memory.percent‚17.443013191223145 network.sent‚ 6.036462e+06 network.recv‚ 1.049264e+06% disk./.usageGB‚1072.4320373535156 disk.rbd0.out‚0 disk.nvme0n1.in‚ 669.43359375% disk.nvme0n1.out‚47873.7197265625 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 2889.67578125$ memory_percent‚98.92973899841309 cpu‚25.262850380621373 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd3.out‚0¯&ø2›Ÿ‚(: ÿ’ãÔêòËÓ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚86.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.271199999999997 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼âŽfL ‚(:Á ÿ’ãÔŸ®ÇÖ disk.sda2.in‚0) disk./.usagePercent‚71.97181897735928& disk.nvme0n1.out‚47874.08935546875# disk.dm-0.out‚47874.08935546875 disk.sda2.out‚0$ memory_percent‚98.92973899841309 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚102.3447265625$ disk./.usageGB‚1072.432113647461 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2857.86328125 disk.dm-1.out‚ 2890.2265625 network.sent‚ 6.041656e+06" disk.nvme0c0n1.in‚ 669.43359375 disk.sda.out‚ 2890.2265625 disk.drbd1777.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd1.out‚ 1.533203125 disk.nvme0n1.in‚ 669.43359375 disk.sda.in‚ 0.07421875' proc.memory.availableMB‚ 175.3515625* proc.memory.percent‚17.443013191223145 cpu‚25.261871777509132 disk.rbd0.out‚0 network.recv‚ 1.052668e+06 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 2890.2265625( disk.nvme0c0n1.out‚47874.08935546875 disk.dm-0.in‚ 669.43359375 disk.rbd2.in‚0ÕŽWá¡‚(:„ Ž“ãÔîô Õ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.449 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç½~±Q¢‚(:Æ Ž“ãÔºì¹× disk.rbd3.in‚0 disk.dm-1.out‚ 2890.98828125 disk.dm-0.in‚ 669.43359375 disk.sda.out‚ 2890.98828125 cpu‚25.260816828673534% disk./.usageGB‚1072.4320373535156 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 2890.98828125 disk.drbd1777.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2857.86328125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 669.43359375( proc.memory.availableMB‚ 175.08984375* proc.memory.percent‚17.443013191223145 network.sent‚ 6.047975e+06 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd0.out‚0$ memory_percent‚98.93133640289307 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚47973.60205078125 disk.sda.in‚ 0.07421875 network.recv‚ 1.057353e+06 disk.sda1.in‚0 disk.rbd2.out‚102.3447265625 disk.nvme0n1.in‚ 669.43359375# disk.dm-0.out‚47973.60205078125) disk./.usagePercent‚71.97181385720857 disk.rbd1.out‚ 1.533203125( disk.nvme0c0n1.out‚47973.60205078125 disk.dm-1.in‚ 0.07421875òâJä2£- _wandbruntime‚3032  _runtime‚3032¦{¼½Ç¤Á _runtime‚3032.419504102 _step‚222 & lr-AdamW/pg1‚0.0002121720544445218 & lr-AdamW/pg2‚0.0002121720544445218 trainer/global_step‚2749 % _timestamp‚1.7883979688596387e+09x Ì¥Æ & lr-AdamW/pg1‚0.0002121720544445218 & lr-AdamW/pg2‚0.0002121720544445218 trainer/global_step‚2749 % _timestamp‚1.7883979688596387e+09 _runtime‚3032.419504102 _step‚222Þn{á;2¦- _wandbruntime‚3032  _runtime‚3032pâå]Ú'§ÔO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚223 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51935033 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72960888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99261582924800 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110554607844352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122841201004544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99261599964160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5018482728448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127917623145984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51935031 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124895921 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122841177064448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5076162895872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72934707 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50200966 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110554356256256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5018482728448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5076422141440 trainer/global_step‚2749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105536107112960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5018249143296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105536125115904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105536125115904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105536107112960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5003109732817 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110554356256256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50200257 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 % _timestamp‚1.7883979688605294e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110554607844352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19737618 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104264692657617 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51935033 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5003338702665 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72960888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 _runtime‚3032.420717449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19736911 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72934707 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127917339960320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5018249143296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51935031 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104264938666825 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30463348 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0'aãDß'¨ÙO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51935033 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72960888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99261582924800 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110554607844352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122841201004544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99261599964160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5018482728448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127917623145984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51935031 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124895921 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122841177064448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5076162895872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72934707 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50200966 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110554356256256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5018482728448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5076422141440 trainer/global_step‚2749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105536107112960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5018249143296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105536125115904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105536125115904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105536107112960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5003109732817 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110554356256256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50200257 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 % _timestamp‚1.7883979688605294e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110554607844352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19737618 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚104264692657617 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51935033 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5003338702665 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72960888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 _runtime‚3032.420717449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19736911 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72934707 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127917339960320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5018249143296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51935031 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104264938666825 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30463348 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚223ß[Ü ’2©- _wandbruntime‚3033  _runtime‚3033¶ßúrªÞM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105573267405824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105573267405824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5077985832448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7883979698655663e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50217996 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51953212 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127962833367552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚110593559296000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5020291890176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99296509618176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105573249402880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104301669093849 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5078253453312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30474001 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124913992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5020291890176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30473999 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72960782 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124940201 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¯Žäìé&«ãM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122884555974144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51953212 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127962833367552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚110593559296000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5020291890176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99296509618176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105573249402880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104301669093849 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5078253453312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30474001 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124913992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5020291890176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30473999 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72960782 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124940201 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105573267405824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105573267405824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5077985832448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7883979698655663e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50217996 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5167797214720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5108720977920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127111973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112501942586880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107393239611904 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5108954563072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106101473598833 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127085790 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20081294 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74241809 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30995998 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚740 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5108954563072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51077290 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112502194174976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107393239611904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52843981 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3085.246247224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52843983 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5167537969152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 _step‚227 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚112501942586880 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130177943704064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125010122549248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107393221608960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52843981 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74267990 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30995996 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107393221608960 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_ba©š%ó/$tch_start/reserved_bytes.all.peak‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52843981 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74267990 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30995996 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107393221608960 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5167797214720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5108720977920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127111973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112501942586880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107393239611904 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5108954563072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106101473598833 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127085790 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20081294 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74241809 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30995998 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚740 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5108954563072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51077290 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112502194174976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107393239611904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52843981 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3085.246247224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52843983 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5167537969152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 _step‚227ã.Š)2»- _wandbruntime‚3086  _runtime‚3086QŽC7å&¼ßM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74294091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31006651 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125053525399040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125053501458944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883980226793725e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107430381901824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74267884 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20088088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127156253 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5110530126336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107430363898880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101042844286976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112541145626624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107430381901824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127130044 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚740 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130222865895424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74267884 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106138204025857 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 _step‚228 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285269504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5169625765888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2799 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127156253 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74294091 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5110763724800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106137958016441 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51094737 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5169364436480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52862160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52862162 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52862162 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112541145626624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5110530126336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52862162 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52862162 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112541145626624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5110530126336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883980226793725e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107430381901824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74267884 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20088088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127156253 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5110530126336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107430363898880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101042844286976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112541145626624 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107430381901824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127130044 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚740 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130222865895424 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74267884 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106138204025857 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 _step‚228 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285269504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5169625765888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚2799 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127156253 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74294091 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5110763724800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261329408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106137958016441 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51094737 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5169364436480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307ä·!ö2¾- _wandbruntime‚3086  _runtime‚3086¥ëõ®Q¿Ë epoch‚2 .loss/FRt2/detection_loss‚2.054746831380404 /loss/FRt2/assignment_loss‚4.422644987473121 /loss/FRt4/assignment_loss‚2.845627348574381 0loss/FBt1/assignment_loss‚0.5815395935460401 0loss/FBt4/assignment_loss‚0.2558334959424018 1loss/SRqqt3/detection_loss‚2.6768996210201927 .loss/FBt2/detection_loss‚4.569372653961182 1loss/SRqqt1/assignment_loss‚3.219001429421561 0loss/SRqqt4/assignment_loss‚2.28239926258507 _step‚229 /loss/FRt3/assignment_loss‚6.008646215711321 1loss/FBt3/assignment_loss‚0.49157178637733523 .loss/FRt3/detection_loss‚2.900655571052006 trainer/global_step‚2799 % _timestamp‚1.7883980226817207e+09 1loss/SRqqt1/detection_loss‚2.6828217052278065 /loss/FBt4/detection_loss‚1.1387146827856116 _runtime‚ 3086.24147725 .loss/FRt1/assignment_loss‚5.54703015089035 0loss/FBt2/assignment_loss‚1.2662398591637611 /loss/FBt3/detection_loss‚2.8668132573366165 1loss/SRqqt2/assignment_loss‚2.458352631992764 2loss/SRqqt3/assignment_loss‚3.3577802958695786 -loss/FRt1/detection_loss‚3.49946216493845 .loss/FRt4/detection_loss‚1.727121629412212 &loss/total_loss‚61.251526811115504 1loss/SRqqt2/detection_loss‚1.4928421487824783 1loss/SRqqt4/detection_loss‚1.2738786190748215 /loss/FBt1/detection_loss‚1.6315308685960441²Â8VÀÐ /loss/FRt3/assignment_loss‚6.008646215711321 1loss/FBt3/assignment_loss‚0.49157178637733523 .loss/FRt3/detection_loss‚2.900655571052006 trainer/global_step‚2799 % _timestamp‚1.7883980226817207e+09 1loss/SRqqt1/detection_loss‚2.6828217052278065 /loss/FBt4/detection_loss‚1.1387146827856116 _runtime‚ 3086.24147725 .loss/FRt1/assignment_loss‚5.54703015089035 0loss/FBt2/assignment_loss‚1.2662398591637611 /loss/FBt3/detection_loss‚2.8668132573366165 1loss/SRqqt2/assignment_loss‚2.458352631992764 2loss/SRqqt3/assignment_loss‚3.3577802958695786 -loss/FRt1/detection_loss‚3.49946216493845 .loss/FRt4/detection_loss‚1.727121629412212 &loss/total_loss‚61.251526811115504 1loss/SRqqt2/detection_loss‚1.4928421487824783 1loss/SRqqt4/detection_loss‚1.2738786190748215 /loss/FBt1/detection_loss‚1.6315308685960441 epoch‚2 .loss/FRt2/detection_loss‚2.054746831380404 /loss/FRt2/assignment_loss‚4.422644987473121 /loss/FRt4/assignment_loss‚2.845627348574381 0loss/FBt1/assignment_loss‚0.5815395935460401 0loss/FBt4/assignment_loss‚0.2558334959424018 1loss/SRqqt3/detection_loss‚2.6768996210201927 .loss/FBt2/detection_loss‚4.569372653961182 1loss/SRqqt1/assignment_loss‚3.219001429421561 0loss/SRqqt4/assignment_loss‚2.28239926258507 _step‚229åÕ÷ò3Á‚(:„ Ê“ãÔñ¿ŽÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚97.775' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.555 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"åUÅ)‚(:È Ê“ãÔ˜ê—Ö disk.rbd3.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2857.86328125 cpu‚25.256939550789227 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.59619140625" disk.nvme0c0n1.in‚ 669.43359375 disk.sda.in‚ 0.07421875sÎÉÛ*) disk./.usagePercent‚71.97232356820925 disk.rbd3.out‚0 disk.nvme0n1.in‚ 669.43359375 disk.rbd1.in‚0 disk.dm-1.out‚ 2897.42578125 disk.sda1.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 669.43359375$ memory_percent‚98.92897605895996 proc.cpu.threads‚17 disk.dm-1.in‚ 0.07421875# disk.dm-0.out‚48080.87158203125 network.recv‚ 1.075987e+06% disk./.usageGB‚1072.4396324157715 disk.drbd1777.in‚0 disk.drbd1777.out‚0' proc.memory.availableMB‚ 175.4765625 disk.sda1.out‚0 disk.sda3.out‚ 2897.42578125 disk.rbd0.out‚0& disk.nvme0n1.out‚48080.87158203125* proc.memory.percent‚17.443013191223145( disk.nvme0c0n1.out‚48080.87158203125 disk.sda.out‚ 2897.42578125 network.sent‚ 6.173641e+06! disk.rbd2.out‚104.09521484375¸È‡MšÃ‚(: Ù“ãÔ¾½Ô gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚77.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚15.491999999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿyoKÄ‚(:À Ù“ãÔ•µ…Ö& disk.nvme0n1.out‚48181.82080078125# proc.memory.rssMB‚ 2857.86328125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.nvme0n1.in‚ 669.43359375 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2903.2578125 network.sent‚ 6.28129e+06 network.recv‚ 1.081981e+06% disk./.usageGB‚1072.4398727416992 disk.sda3.in‚ 0.07421875( disk.nvme0c0n1.out‚48181.82080078125# disk.dm-0.out‚48181.82080078125# memory_percent‚98.9307165145874 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 2903.2578125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.59619140625 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 104.181640625 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 669.43359375 disk.drbd1777.in‚0 disk.dm-1.out‚ 2903.2578125 disk.rbd0.out‚0* proc.memory.percent‚17.443013191223145( disk./.usagePercent‚71.9723396966839 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 669.43359375( proc.memory.availableMB‚ 175.19140625 cpu‚25.25603071241081 disk.rbd3.in‚0Òñ Å‚(:… è“ãÔû†ýÒ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚91.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠO–ŠJÆ‚(:¿ è“ãÔ—†äÕ disk.drbd1777.out‚0# proc.memory.rssMB‚ 2857.86328125% disk./.usageGB‚1072.4396324157715 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875 proc.cpu.threads‚17) disk./.usagePercent‚71.97232356820925 disk.sda3.out‚ 2903.92578125 disk.nvme0n1.in‚ 669.43359375 disk.dm-1.out‚ 2903.92578125 disk.sda.out‚ 2903.92578125 disk.rbd3.in‚0 cpu‚25.25513251891763 network.sent‚ 6.286728e+06 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 175.19140625 disk.rbd0.in‚0$ disk.nvme0n1.out‚48182.658203125! disk.dm-0.out‚48182.658203125 disk.sda1.in‚0 disk.drbd1777.in‚0# memory_percent‚98.9307165145874& disk.nvme0c0n1.out‚48182.658203125 network.recv‚ 1.085819e+06 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0 disk.rbd1.out‚ 1.59619140625" disk.nvme0c0n1.in‚ 669.43359375 disk.dm-0.in‚ 669.43359375* proc.memory.percent‚17.443013191223145 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd2.out‚ 104.197265625.ùz»Ç‚(:… ÷“ãÔˆ¼ÝÓ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚89.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9178 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'Û\3È‚(:¨ ÷“ãÔò†ÕÕ% disk./.usageGB‚1072.4394721984863$ disk.nvme0n1.out‚48183.025390625 disk.sda3.out‚2904.5& disk.nvme0c0n1.out‚48183.025390625$ memory_percent‚98.93226623535156 network.recv‚ 1.089497e+06 disk.rbd0.out‚0 disk.sda.out‚2904.5 disk.rbd3.in‚0* proc.memory.percent‚17.443013191223145 disk.dm-0.in‚ 669.43359375 network.sent‚ 6.291895e+06 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚2904.5! disk.dm-0.out‚48183.025390625 disk.rbd1.out‚ 1.59619140625 disk.sda.in‚ 0.07421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 669.43359375# proc.memory.rssMB‚ 2857.86328125 proc.cpu.threads‚17) disk./.usagePercent‚71.97231281589282 disk.rbd2.out‚ 104.322265625 disk.sda3.in‚ 0.07421875 disk.nvme0n1.in‚ 669.43359375 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ proc.memory.availableMB‚174.9375 cpu‚25.254083461488552`8(2É- _wandbruntime‚3137  _runtime‚3137JÑÂvÈÊ ' lr-AdamW/pg1‚0.00021074551571047937 ' lr-AdamW/pg2‚0.00021074551571047937 trainer/global_step‚2849 $ _timestamp‚1.788398073845294e+09 _runtime‚3137.405126698 _step‚230+ ÖæÍËÇ ' lr-AdamW/pg1‚0.00021074551571047937 ' lr-AdamW/pg2‚0.00021074551571047937 trainer/global_step‚2849 $ _timestamp‚1.788398073845294e+09 _runtime‚3137.405126698 _step‚230æü¡¯÷2Ì- _wandbruntime‚3137  _runtime‚31371šZ&Û'ÍÕO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51958428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚754 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109250354107904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53752931 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5183739229081 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5199426397696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75575092 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293671424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132438019745792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127179068034048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269731328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 _runtime‚ 3137.40615172 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5199192812544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109250354107904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _step‚231 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129328025 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53752933 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5258951711744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75548911 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5259221443072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127179091974144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102754269301760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53752933 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20429780 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938008530841 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53752931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5199426397696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114449528917504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75548911 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚2849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114449528917504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75575092 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5183510259233 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109250336104960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5199192812544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31528646 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129301842 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 ÄM€à'ÎÚO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5258951711744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75548911 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5259221443072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127179091974144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102754269301760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53752933 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20429780 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938008530841 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53752931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5199426397696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114449528917504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75548911 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚2849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114449528917504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75575092 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5183510259233 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109250336104960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5199192812544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31528646 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129301842 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51958428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚754 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109250354107904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53752931 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5183739229081 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5199426397696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75575092 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293671424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132438019745792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127179068034048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269731328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 _runtime‚ 3137.40615172 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5199192812544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109250354107904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _step‚231 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129328025 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53752933ç8»Q2Ï- _wandbruntime‚3138  _runtime‚3138eF®ÐàM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20436691 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114488731957248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚749 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20435942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109287478394880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132483512666624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75601193 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51975992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51975241 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5185313992673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75601193 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 trainer/global_step‚2849 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5201001960960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114488731957248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102789195995136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883980748514218e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109287496397824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129372305 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102789178955776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53771112 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75574986 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114488480355840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5261041782784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53771110 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5185542962729 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31539301 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5201235559424 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109287478394880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53771112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 _runtime‚3138.412167806æ1s¤ë&ÑåM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53771112 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75574986 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114488480355840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5261041782784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53771110 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5185542962729 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31539301 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5201235559424 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109287478394880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53771112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 _runtime‚3138.412167806 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20436691 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114488731957248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚749 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20435942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109287478394880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132483512666624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75601193 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51975992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51975241 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5185313992673 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75601193 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 trainer/global_step‚2849 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5201001960960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114488731957248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102789195995136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883980748514218e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109287496397824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129372305 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102789178955776èÇ“å·2Ò- _wandbruntime‚3138  _runtime‚3138Á^É`SÓÍ 0loss/FRt4/assignment_loss‚2.2516283351799538 0loss/FBt2/assignment_loss‚1.3971885293722153 /loss/FBt3/detection_loss‚3.9024897694587706 /loss/FRt3/assignment_loss‚6.535946740044488 1loss/SRqqt4/assignment_loss‚2.051040600479385 0loss/FBt1/assignment_loss‚0.7154675888111858 .loss/FRt3/detection_loss‚2.778621382183499 0loss/SRqqt2/detection_loss‚1.478559468423619 _step‚233 /loss/SRqqt1/detection_loss‚2.61041226263704 /loss/FRt2/detection_loss‚2.1732818871213677 /loss/FRt4/detection_loss‚1.9971224090148663 trainer/global_step‚2849 1loss/SRqqt2/assignment_loss‚2.302167797088623 2loss/SRqqt3/assignment_loss‚2.7379214107990264 /loss/FBt4/detection_loss‚1.0845025020762016 _runtime‚3138.413486542 1loss/FBt4/assignment_loss‚0.31342551383061046 .loss/FRt1/detection_loss‚2.562665281387476 1loss/SRqqt3/detection_loss‚3.1363730877637863 .loss/FBt1/detection_loss‚1.409167503503303 epoch‚2 /loss/FRt2/assignment_loss‚3.934613112817731 1loss/SRqqt1/assignment_loss‚2.910189965675617 0loss/FBt3/assignment_loss‚0.5433207255118759 1loss/SRqqt4/detection_loss‚1.1570477585457726 .loss/FBt2/detection_loss‚2.770402044057846 %loss/total_loss‚58.93950346079496 /loss/FRt1/assignment_loss‚6.185947785010705 % _timestamp‚1.7883980748537438e+09Ñ.eXÔÒ /loss/FRt3/assignment_loss‚6.535946740044488 1loss/SRqqt4/assignment_loss‚2.051040600479385 0loss/FBt1/assignment_loss‚0.7154675888111858 .loss/FRt3/detection_loss‚2.778621382183499 0loss/SRqqt2/detection_loss‚1.478559468423619 _step‚233 /loss/SRqqt1/detection_loss‚2.61041226263704 /loss/FRt2/detection_loss‚2.1732818871213677 /loss/FRt4/detection_loss‚1.9971224090148663 trainer/global_step‚2849 1loss/SRqqt2/assignment_loss‚2.302167797088623 2loss/SRqqt3/assignment_loss‚2.7379214107990264 /loss/FBt4/detection_loss‚1.0845025020762016 _runtime‚3138.413486542 1loss/FBt4/assignment_loss‚0.31342551383061046 .loss/FRt1/detection_loss‚2.562665281387476 1loss/SRqqt3/detection_loss‚3.1363730877637863 .loss/FBt1/detection_loss‚1.409167503503303 epoch‚2 /loss/FRt2/assignment_loss‚3.934613112817731 1loss/SRqqt1/assignment_loss‚2.910189965675617 0loss/FBt3/assignment_loss‚0.5433207255118759 1loss/SRqqt4/detection_loss‚1.1570477585457726 .loss/FBt2/detection_loss‚2.770402044057846 %loss/total_loss‚58.93950346079496 /loss/FRt1/assignment_loss‚6.185947785010705 % _timestamp‚1.7883980748537438e+09 0loss/FRt4/assignment_loss‚2.2516283351799538 0loss/FBt2/assignment_loss‚1.3971885293722153 /loss/FBt3/detection_loss‚3.9024897694587706é)Ê"=Õ‚(:… †”ãÔ§¬ýÒ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8842 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÿ?IÖ‚(:¾ †”ãÔñÀÖ% disk.nvme0n1.out‚48282.3134765625% disk./.usageGB‚1072.4403266906738 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 669.43359375 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.07421875 disk.drbd1777.out‚0' disk.nvme0c0n1.out‚48282.3134765625 cpu‚25.253112519808194 network.sent‚ 6.298367e+06 network.recv‚ 1.094314e+06 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 2905.703125 disk.rbd0.in‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 2905.703125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.59619140625* proc.memory.percent‚17.443013191223145) disk./.usagePercent‚71.97237016158049 disk.sda.in‚ 0.07421875 disk.rbd2.out‚ 105.912109375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 669.43359375 disk.dm-0.in‚ 669.43359375" disk.dm-0.out‚48282.3134765625$ memory_percent‚98.93100261688232 disk.sda.out‚ 2905.703125( proc.memory.availableMB‚ 175.14453125# proc.memory.rssMB‚ 2857.86328125Q"•‘ׂ(:† •”ãÔˆ”Ó gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚102.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5226 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä“HTLØ‚(:Á •”ãÔŒ¸× disk.rbd2.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 175.3828125 network.sent‚ 6.406561e+06 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 2907.51953125# memory_percent‚98.9295482635498 disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 669.43359375 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875' disk.nvme0c0n1.out‚48282.7880859375 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.59619140625# proc.memory.rssMB‚ 2857.86328125 disk.rbd0.in‚0% disk.nvme0n1.out‚48282.7880859375" disk.dm-0.out‚48282.7880859375 disk.sda.in‚ 0.07421875 disk.nvme0n1.in‚ 669.43359375 disk.sda1.in‚0 network.recv‚ 1.100781e+06) disk./.usagePercent‚71.97235403310583 disk.rbd1.in‚0* proc.memory.percent‚17.443013191223145 cpu‚25.252212786669944 proc.cpu.threads‚17$ disk./.usageGB‚1072.440086364746 disk.dm-1.out‚ 2907.51953125 disk.rbd0.out‚0 disk.sda.out‚ 2907.51953125 disk.sda1.out‚0 disk.rbd2.out‚ 105.974609375" disk.nvme0c0n1.in‚ 669.43359375©=J›Ù‚(: ¤”ãÔÛ½‡Ó gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.245' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.049000000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á„¥>PÚ‚(:Å ¤”ãÔÿçÕ network.sent‚ 6.412798e+06) disk./.usagePercent‚71.97229131125994 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0 disk.sda2.out‚0& disk.nvme0n1.out‚48382.38525390625# proc.memory.rssMB‚ 2857.86328125" disk.nvme0c0n1.in‚ 669.43359375 disk.rbd0.in‚0' proc.memory.availableMB‚ 175.5078125$ disk./.usageGB‚1072.439151763916 disk.sda1.in‚0 disk.drbd1777.in‚0( disk.nvme0c0n1.out‚48382.38525390625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 669.43359375# disk.dm-0.out‚48382.38525390625 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.443013191223145 cpu‚25.251293539490653 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 2908.35546875 disk.dm-1.in‚ 0.07421875 network.recv‚ 1.105211e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 669.43359375 proc.cpu.threads‚17 disk.rbd1.out‚ 1.59619140625 disk.sda.out‚ 2908.35546875 disk.rbd2.in‚0 disk.rbd2.out‚ 105.990234375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 2908.35546875$ memory_percent‚98.92878532409668Dˆ"62Û- _wandbruntime‚3189  _runtime‚3189ÍuÐZÈÜ ' lr-AdamW/pg1‚0.00021000114920608222 ' lr-AdamW/pg2‚0.00021000114920608222 trainer/global_step‚2899 $ _timestamp‚1.788398125981687e+09 _runtime‚3189.541533128 _step‚234G½?­ÍÝÇ ' lr-AdamW/pg1‚0.00021000114920608222 ' lr-AdamW/pg2‚0.00021000114920608222 trainer/global_step‚2899 $ _timestamp‚1.788398125981687e+09 _runtime‚3189.541533128 _step‚234ê؉é2Þ- _wandbruntime‚3189  _runtime‚3189üѼÜ'ßÖO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32061296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54661881 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134698659398144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3189.542610659 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271828480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚116397366836224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32061298 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54661883 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54661883 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚752 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5289898232320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131544077 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295768576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104500603970560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5273710522441 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 _step‚235 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52832472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20771174 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54661881 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 trainer/global_step‚2899 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5350350110720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104500586931200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116397115248128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111107468603904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5289898232320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5289664647168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109774297453641 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111107468603904 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109774543462849 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5350621939200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76882194 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 % _timestamp‚1.7883981259825482e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111107450600960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134698363629568 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116397115248128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76882194 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 % _timestamp‚1.7883981259825482e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111107450600960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134698363629568 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116397115248128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134698659398144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3189.542610659 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271828480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚116397366836224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32061298 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54661883 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54661883 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚752 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504ët8'2á- _wandbruntime‚3190  _runtime‚3190’N}Ræ&âàM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5291473795584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76908295 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54680062 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54680060 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131588357 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5352439310336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚718 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129391416368640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5352180078080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76908295 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111144610893824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 % _timestamp‚1.7883981269873068e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116436318287872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111144610893824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76882088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5275743225937 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131562148 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚109811273889873 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5275514255881 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116436066686464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32071951 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131588357 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116436318287872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52850156 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111144592890880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20777489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134743572506624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 trainer/global_step‚2899 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76882088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5291707394048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104535513624576 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131562148 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109811027880457 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20778205 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54680060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚716 _step‚236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5291473795584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3190.548022218 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52849438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32071949 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129391392428544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5291707394048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104535530663936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131588357 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5352439310336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚718 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129391416368640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5352180078080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76908295 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111144610893824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 % _timestamp‚1.7883981269873068e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116436318287872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111144610893824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76882088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5275743225937 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131562148 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚109811273889873 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5275514255881 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116436066686464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32071951 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131588357 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116436318287872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52850156 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111144592890880 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20777489 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134743572506624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 trainer/global_step‚2899 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76882088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5291707394048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104535513624576 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131562148 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109811027880457 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20778205 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54680060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚716 _step‚236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5291473795584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚3190.548022218 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52849438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32071949 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129391392428544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5291707394048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104535530663936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136958721358336 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32593948 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5363910785649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133733946 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78163115 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118344701578752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112964565096960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55570833 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21117909 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55570831 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5380370066944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106246921600000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131516982943744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112964583099904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5380136481792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚738 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5441762354688 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136959004544000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111611078394857 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 _runtime‚3241.788601281 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚740 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111610832385649 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78189296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21117171 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚118344953166848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133760129 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53711857 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5380370066944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚118344953166848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 $ _timestamp‚1.788398178228401e+09 _step‚239 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112964565096960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55570833 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78189296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5364139755497 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131516959003648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2949–»Öà'ôÚO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136959004544000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111611078394857 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 _runtime‚3241.788601281 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚740 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111610832385649 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78189296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21117171 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚118344953166848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133760129 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53711857 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5380370066944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚118344953166848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 $ _timestamp‚1.788398178228401e+09 _step‚239 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112964565096960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55570833 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78189296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5364139755497 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131516959003648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2949 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚118344701578752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55570831 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53711117 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5442021600256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136958721358336 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32593948 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5363910785649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133733946 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78163115 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118344701578752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112964565096960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55570833 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21117909 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55570831 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5380370066944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106246921600000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131516982943744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112964583099904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5380136481792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚738 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5441762354688 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133778200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32604601 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5365714519089 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53729173 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131560361853440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113001707386880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113001725389824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 trainer/global_step‚2949 $ _timestamp‚1.788398179197912e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804409 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5382179228672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53728458 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106281865332736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78189190 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32604599 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137003929285632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137004212457984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55589012 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78215397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚118383904618496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5365943489145 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 E=Devicap‰1 eStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55589010 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78189190 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131560337913344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106281848293376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113001707386880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113001725389824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5381945630208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21123859 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5381945630208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚118383653017088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 3242.75841495 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21124572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133778200 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78215397 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0žÆçé&÷ãM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5382179228672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118383904618496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804409 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5382179228672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53728458 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106281865332736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78189190 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32604599 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137003929285632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137004212457984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55589012 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78215397 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚118383904618496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5365943489145 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55589010 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78189190 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131560337913344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106281848293376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113001707386880 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113001725389824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5381945630208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21123859 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5381945630208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚118383653017088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 3242.75841495 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21124572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133778200 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78215397 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133778200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32604601 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5365714519089 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53729173 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131560361853440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113001707386880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113001725389824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 trainer/global_step‚2949 $ _timestamp‚1.788398179197912e+09 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5454340018705 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56479781 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135976181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120292539497472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113447613326865 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79496398 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113447367317657 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56479781 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139219331802112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 _runtime‚3293.229143908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107993273308160 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21462415 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚2999 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120292539497472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135949998 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114821697595904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5470608316416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56479783 $ _timestamp‚1.788398229669064e+09 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120292287909376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133685904488448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79470217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114821697595904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114821679592960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133685928428544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 _step‚243 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120292287909376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚107993256268800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56479783 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79470217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54589013 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33126598 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5533144128000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5470608316416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54588281 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5533403373568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0,ÿS†ÚO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5454111048857 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inact_W†ÒÊ"ive_split.small_pool.allocated‚21462415 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚2999 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120292539497472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135949998 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114821697595904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5470608316416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56479783 $ _timestamp‚1.788398229669064e+09 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120292287909376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133685904488448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79470217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114821697595904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114821679592960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133685928428544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 _step‚243 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120292287909376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚107993256268800 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56479783 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79470217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54589013 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33126598 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5533144128000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5470608316416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54588281 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5533403373568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114821679592960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5470841901568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21461685 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5454340018705 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56479781 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135976181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120292539497472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113447613326865 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79496398 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113447367317657 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56479781 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139219331802112 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 _runtime‚3293.229143908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107993273308160 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883982306580865e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5472651063296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5535232391680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133729283398144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5456143752353 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136020461 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5472417464832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136020461 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚721 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108028200001536 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108028182962176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114858821882880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120331239347712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135994252 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133729307338240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120331490949120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56497960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139264539729920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _runtime‚3294.218577842 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79522499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21469571 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5534966867968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33137249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56497962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54606822 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120331490949120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139264250266112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113484097744473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _step‚244 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79522499 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79496292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114858839885824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114858839885824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56497962 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5472417464832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 trainer/global_step‚2999 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54606101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33137251åBë&‰åM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5472651063296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56497960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139264539729920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _runtime‚3294.218577842 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79522499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21469571 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5534966867968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33137249 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56497962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54606822 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120331490949120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139264250266112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113484097744473 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _step‚244 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79522499 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79496292 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114858839885824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114858839885824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56497962 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5472417464832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 trainer/global_step‚2999 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54606101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33137251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79496292 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883982306580865e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5472651063296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5535232391680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133729283398144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5456143752353 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136020461 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5472417464832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136020461 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚721 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108028200001536 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108028182962176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114858821882880 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120331239347712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135994252 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133729307338240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120331490949120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5561313736192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116678812091904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57388731 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116678794088960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116678794088960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57388731 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5544311312065 _step‚247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33659248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624517936128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116678812091904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 % _timestamp‚1.7883982821081529e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5561313736192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141479651095040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55468506 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122240125828096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135854873913344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122239874240000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33659246 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0oÙqá'šÛO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80803500 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5561080151040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80777319 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109739607976960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135854849973248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624777181696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚756 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5561313736192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116678812091904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57388731 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116678794088960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116678794088960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57388731 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5544311312065 _step‚247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33659248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624517936128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116678812091904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 % _timestamp‚1.7883982821081529e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5561313736192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141479651095040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55468506 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122240125828096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135854873913344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122239874240000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33659246 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5561080151040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21809258 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115284148258873 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57388733 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚758 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5544540281913 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138166050 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122240125828096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122239874240000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80803500 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55467748 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138192233 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141479367909376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115283902249665 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80777319 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _runtime‚3345.668268976÷0-µ^2›- _wandbruntime‚3346  _runtime‚3346â>,F#œßM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80803394 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109774534670336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116715936378880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546115045505 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138210304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5563122897920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55485050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33669901 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80829601 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21815151 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788398283098725e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135898252823040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚3049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141524862556672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115320878685897 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115320632676481 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141524581481472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5626609733632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257135104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5562889299456 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138236513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138236513 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138210304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚721 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116715936378880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116715954381824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55485771 ]KDeviceStatsMonitor.on_train_batch_end/iaKv nactive_split_bytes.small_pool.freed‚ 5626352598528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57406910 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116715954381824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546344015561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57406912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3346.659499249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135898228882944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21815870 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80829601 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5563122897920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80803394 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33669899 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5562889299456 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57406912 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109774534670336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116715936378880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5546115045505 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138210304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5563122897920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55485050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33669901 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80829601 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21815151 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788398283098725e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135898252823040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚3049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141524862556672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115320878685897 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115320632676481 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141524581481472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5626609733632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 257135104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5562889299456 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138236513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138236513 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138210304 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚721 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116715936378880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116715954381824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55485771 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5626352598528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57406910 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116715954381824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546344015561 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57406912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3346.659499249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135898228882944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21815870 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80829601 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5563122897920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80803394 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33669899 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960ø¬´2ž- _wandbruntime‚3346  _runtime‚3346¨’ôYŸÓ % _timestamp‚1.7883982831009414e+09 2loss/SRqqt1/assignment_loss‚2.6516686402834377 _runtime‚3346.660733075 .loss/FRt3/detection_loss‚2.601826749032452 1loss/SRqqt3/detection_loss‚3.3060808295295354 _step‚249 .loss/FRt2/assignment_loss‚4.23647849872464 /loss/FBt2/detection_loss‚3.9718874394893646 trainer/global_step‚3049 1loss/FBt4/assignment_loss‚0.22743091750651895 epoch‚2 /loss/FRt3/assignment_loss‚5.917034103756859 2loss/SRqqt2/assignment_loss‚2.5506073189409157 2loss/SRqqt4/assignment_loss‚2.0842326296047426 1loss/SRqqt2/detection_loss‚1.4758525589775098 /loss/FBt1/detection_loss‚1.5451456318641532 /loss/FBt4/detection_loss‚1.0871386670253493 %loss/total_loss‚58.51836718990421 /loss/FRt1/assignment_loss‚6.191438416639964 0loss/FRt4/assignment_loss‚2.5300189753373465 2loss/SRqqt3/assignment_loss‚2.4082689994857427 0loss/FBt3/assignment_loss‚0.8484815487441105 /loss/FRt1/detection_loss‚2.6042803724606833 0loss/FBt1/assignment_loss‚0.7105014808456214 -loss/FRt2/detection_loss‚2.06311047272604 /loss/FRt4/detection_loss‚1.8631899130565148 1loss/SRqqt1/detection_loss‚2.7244946154264302 /loss/FBt3/detection_loss‚2.4142656188744764 0loss/FBt2/assignment_loss‚1.3965301513671875 1loss/SRqqt4/detection_loss‚1.1084026402046143ud^ Ø /loss/FRt1/assignment_loss‚6.191438416639964 0loss/FRt4/assignment_loss‚2.5300189753373465 2loss/SRqqt3/assignment_loss‚2.4082689994857427 0loss/FBt3/assignment_loss‚0.8484815487441105 /loss/FRt1/detection_loss‚2.6042803724606833 0loss/FBt1/assignment_loss‚0.7105014808456214 -loss/FRt2/detection_loss‚2.06311047272604 /loss/FRt4/detection_loss‚1.8631899130565148 1loss/SRqqt1/detection_loss‚2.7244946154264302 /loss/FBt3/detection_loss‚2.4142656188744764 0loss/FBt2/assignment_loss‚1.3965301513671875 1loss/SRqqt4/detection_loss‚1.1084026402046143 % _timestamp‚1.7883982831009414e+09 2loss/SRqqt1/assignment_loss‚2.6516686402834377 _runtime‚3346.660733075 .loss/FRt3/detection_loss‚2.601826749032452 1loss/SRqqt3/detection_loss‚3.3060808295295354 _step‚249 .loss/FRt2/assignment_loss‚4.23647849872464 /loss/FBt2/detection_loss‚3.9718874394893646 trainer/global_step‚3049 1loss/FBt4/assignment_loss‚0.22743091750651895 epoch‚2 /loss/FRt3/assignment_loss‚5.917034103756859 2loss/SRqqt2/assignment_loss‚2.5506073189409157 2loss/SRqqt4/assignment_loss‚2.0842326296047426 1loss/SRqqt2/detection_loss‚1.4758525589775098 /loss/FBt1/detection_loss‚1.5451456318641532 /loss/FBt4/detection_loss‚1.0871386670253493 %loss/total_loss‚58.51836718990421ù}Á±s›¡‚(: Ø•ãÔ©“€Ó gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚96.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.261799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢1öH¢‚(:½ Ø•ãÔŒ¾ÜÕ disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 6.788874e+06 network.recv‚ 1.161273e+06 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.59619140625% disk./.usageGB‚1072.4394569396973 disk.dm-1.in‚ 0.07421875 disk.sda.out‚ 2956.296875 proc.cpu.threads‚17 disk.rbd1.in‚0' disk.nvme0c0n1.out‚48784.5537109375 disk.nvme0n1.in‚ 669.43359375 disk.sda2.in‚0# memory_percent‚98.9316463470459' proc.memory.availableMB‚ 175.0390625* proc.memory.percent‚17.443060874938965 disk.sda3.out‚ 2956.296875" disk.nvme0c0n1.in‚ 669.43359375 disk.dm-1.out‚ 2956.296875" disk.dm-0.out‚48784.5537109375 disk.sda.in‚ 0.07421875 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚71.97231179186268 disk.rbd0.in‚0% disk.nvme0n1.out‚48784.5537109375 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2857.87109375 disk.sda2.out‚0 cpu‚25.24089119787203 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚109.82373046875 disk.dm-0.in‚ 669.43359375ÙLI£‚(:… ç•ãÔ‘¶õÒ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1324 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆóÎK¤‚(:À ç•ãÔÔàÈÕ! disk.rbd2.out‚109.94873046875 disk.rbd3.in‚0& disk.nvme0n1.out‚48883.99072265625 disk.sda.out‚ 2956.828125* proc.memory.percent‚17.443060874938965 disk.sda.in‚ 0.07421875 disk.sda2.out‚0 network.recv‚ 1.167115e+06% disk./.usageGB‚1072.4395637512207 disk.rbd0.in‚0# disk.dm-0.out‚48883.99072265625 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875( disk.nvme0c0n1.out‚48883.99072265625 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2956.828125 disk.rbd0.out‚0 disk.dm-0.in‚ 669.43359375 disk.drbd1777.in‚0 network.sent‚ 6.896377e+06) disk./.usagePercent‚71.97231896007364 disk.rbd1.out‚ 1.59619140625 disk.nvme0n1.in‚ 669.43359375 disk.sda2.in‚0% proc.memory.availableMB‚ 175.28125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2857.87109375 cpu‚25.240125347400426 disk.rbd2.in‚0 disk.sda3.out‚ 2956.828125" disk.nvme0c0n1.in‚ 669.43359375 disk.drbd1777.out‚0$ memory_percent‚98.93016815185547g-~B¥‚(:… ö•ãÔ½÷Ò gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.133' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8266 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÑ3©B¦‚(:· ö•ãÔŒ×©Ö disk.rbd3.in‚0 disk.sda2.in‚0) disk./.usagePercent‚71.97230999980994" disk.dm-0.out‚48884.7373046875 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 2957.6875 disk.rbd0.in‚0 disk.sda3.in‚ 0.07421875 proc.cpu.threads‚17 disk.rbd1.out‚ 1.59619140625 disk.dm-0.in‚ 669.43359375 disk.sda.out‚ 2957.6875 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0% disk./.usageGB‚1072.4394302368164 disk.sda.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875$ memory_percent‚98.93169403076172# proc.memory.rssMB‚ 2857.87109375 cpu‚25.239279082059564 network.recv‚ 1.171591e+06% proc.memory.availableMB‚ 175.03125 disk.nvme0n1.in‚ 669.43359375 disk.sda1.in‚0! disk.rbd2.out‚109.96435546875" disk.nvme0c0n1.in‚ 669.43359375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚48884.7373046875% disk.nvme0n1.out‚48884.7373046875 disk.rbd2.in‚0 disk.sda3.out‚ 2957.6875* proc.memory.percent‚17.443060874938965 network.sent‚ 6.902691e+06 disk.rbd0.out‚0©oÒ2§- _wandbruntime‚3397  _runtime‚3397&4¡É¨Ã trainer/global_step‚3099 % _timestamp‚1.7883983341436322e+09 _runtime‚3397.703511024 _step‚250 ' lr-AdamW/pg1‚0.00020682003811677588 ' lr-AdamW/pg2‚0.000206820038116775881ÖW\Î©È ' lr-AdamW/pg1‚0.00020682003811677588 ' lr-AdamW/pg2‚0.00020682003811677588 trainer/global_step‚3099 % _timestamp‚1.7883983341436322e+09 _runtime‚3397.703511024 _step‚250ú+›C¬2ª- _wandbruntime‚3397  _runtime‚3397}n©àÜ'«ÖO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 278991360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56343277 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117120683190881 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58297683 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚721 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5634511575273 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 _step‚251 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34191898 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124187712158720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118535908584960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5651785570816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82110602 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚723 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82084421 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22152102 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58297681 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 _runtime‚3397.704551273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140382102 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5651785570816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140408285 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111485925606400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 % _timestamp‚1.7883983341445277e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118535908584960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5715978475008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118535926587904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124187712158720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124187460570624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 255051264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82084421 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34191896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22151381 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111485942645760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5651551985664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5634740545121 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 trainer/global_step‚3099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5651551985664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143739773933056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58297681 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117120437181673 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58297683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138023819398144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56344000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143740052924416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5716233526272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118535926587904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5715978475008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118535926587904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124187712158720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124187460570624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 255051264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on¯&,T"_train_batch_start/allocation.small_pool.freed‚82084421 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34191896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22151381 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111485942645760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5651551985664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5634740545121 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 trainer/global_step‚3099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5651551985664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143739773933056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58297681 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117120437181673 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58297683 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138023819398144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56344000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143740052924416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5716233526272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118535926587904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117120683190881 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58297683 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚721 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5634511575273 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 _step‚251 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34191898 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124187712158720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118535908584960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5651785570816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82110602 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚723 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82084421 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22152102 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58297681 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 _runtime‚3397.704551273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140382102 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5651785570816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140408285 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0û`7 Z2­- _wandbruntime‚3398  _runtime‚3398O´Jæ&®àM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140426356 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82136703 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289463808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22158308 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143784976807936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138067198307840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58315860 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140452565 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5636544278769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82110496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5653594732544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34202551 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚731 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111520852299776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118573050874880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5653361134080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22159039 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82136703 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5717802440192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 _runtime‚3398.685857819 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚124226663610368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118573068877824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5653594732544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58315862 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117157413617905 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111520869339136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 _step‚252 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58315860 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 trainer/global_step‚3099 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚124226412008960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58315862 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138067174367744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 % _timestamp‚1.7883983351248815e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82110496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56360857 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5718067963904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56361590 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5636315308713 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 % _timestamp‚1.7883983351248815e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82110496 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56360857 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5718067963904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56361590 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5636315308713 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22158308 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143784976807936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138067198307840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58315860 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140452565 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5636544278769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82110496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5653594732544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34202551 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚731 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111520852299776 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118573050874880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5653361134080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22159039 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82136703 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5717802440192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 _runtime‚3398.685857819 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚124226663610368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118573068877824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5653594732544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58315862 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117157413617905 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111520869339136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 _step‚252 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58315860üŸú¼2°- _wandbruntime‚3398  _runtime‚3398yÐkS±Í 1loss/SRqqt2/assignment_loss‚2.615007358044386 2loss/SRqqt3/assignment_loss‚2.6581650495529177 1loss/FBt2/assignment_loss‚0.24181300126074348 -loss/FRt2/detection_loss‚2.12367775480626 %loss/total_loss‚55.25141152654031 epoch‚2 0loss/FBt1/assignment_loss‚0.6477223975229824 /loss/FBt1/detection_loss‚1.4311421746905055 trainer/global_step‚3099 /loss/FRt2/assignment_loss‚4.137945120617495 /loss/FRt4/assignment_loss‚2.584970566461671 2loss/SRqqt1/assignment_loss‚2.7186748307684194 .loss/FRt3/detection_loss‚2.452674553837887 /loss/FRt1/assignment_loss‚6.138530080968684 1loss/SRqqt4/assignment_loss‚2.301656818431285 1loss/SRqqt3/detection_loss‚2.8593210315704347 $ _timestamp‚1.788398335127813e+09 /loss/FRt3/assignment_loss‚5.732522110606348 .loss/FRt1/detection_loss‚2.165301415053281 /loss/FRt4/detection_loss‚1.8556451606300641 /loss/FBt2/detection_loss‚2.4275859594345093 /loss/FBt3/detection_loss‚2.7729538570750845 0loss/FBt3/assignment_loss‚0.5829953021725074 0loss/SRqqt4/detection_loss‚1.326973830246263 0loss/FBt4/assignment_loss‚0.2980784469924417 0loss/SRqqt1/detection_loss‚2.705715832502946 1loss/SRqqt2/detection_loss‚1.2442507315427065 /loss/FBt4/detection_loss‚1.2280881417504812 _runtime‚3398.687614538 _step‚253e^jäX²Ò $ _timestamp‚1.788398335127813e+09 /loss/FRt3/assignment_loss‚5.732522110606348 .loss/FRt1/detection_loss‚2.165301415053281 /loss/FRt4/detection_loss‚1.8556451606300641 /loss/FBt2/detection_loss‚2.4275859594345093 /loss/FBt3/detection_loss‚2.7729538570750845 0loss/FBt3/assignment_loss‚0.5829953021725074 0loss/SRqqt4/detection_loss‚1.326973830246263 0loss/FBt4/assignment_loss‚0.2980784469924417 0loss/SRqqt1/detection_loss‚2.705715832502946 1loss/SRqqt2/detection_loss‚1.2442507315427065 /loss/FBt4/detection_loss‚1.2280881417504812 _runtime‚3398.687614538 _step‚253 1loss/SRqqt2/assignment_loss‚2.615007358044386 2loss/SRqqt3/assignment_loss‚2.6581650495529177 1loss/FBt2/assignment_loss‚0.24181300126074348 -loss/FRt2/detection_loss‚2.12367775480626 %loss/total_loss‚55.25141152654031 epoch‚2 0loss/FBt1/assignment_loss‚0.6477223975229824 /loss/FBt1/detection_loss‚1.4311421746905055 trainer/global_step‚3099 /loss/FRt2/assignment_loss‚4.137945120617495 /loss/FRt4/assignment_loss‚2.584970566461671 2loss/SRqqt1/assignment_loss‚2.7186748307684194 .loss/FRt3/detection_loss‚2.452674553837887 /loss/FRt1/assignment_loss‚6.138530080968684 1loss/SRqqt4/assignment_loss‚2.301656818431285 1loss/SRqqt3/detection_loss‚2.8593210315704347ýXXª³‚(:… …–ãÔŠ©Ó gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚81.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.3238 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%¥¾ý´‚(:· …–ãÔžœÓÖ network.sent‚ 6.908217e+06 disk.rbd1.in‚0& disk.nvme0c0n1.out‚48983.990234375 disk.rbd0.in‚0 disk.dm-0.in‚ 669.43359375 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.sda3.out‚ 2958.16796875* proc.memory.percent‚17.443060874938965 disk.sda1.in‚0 disk.drbd1777.in‚0" disk.nvme0c0n1.in‚ 669.43359375 proc.memory.availableMB‚175 cpu‚25.238519597829512 disk.sda.out‚ 2958.16796875 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2958.16796875 disk.rbd0.out‚0 proc.cpu.threads‚17! disk.dm-0.out‚48983.990234375! disk.rbd2.out‚111.66357421875 disk.sda2.out‚0# proc.memory.rssMB‚ 2857.87109375) disk./.usagePe L¿…Ercent‚71.97234814493254$ disk./.usageGB‚1072.439998626709 disk.nvme0n1.in‚ 669.43359375$ disk.nvme0n1.out‚48983.990234375 disk.sda2.in‚0 disk.rbd1.out‚ 1.59619140625 network.recv‚ 1.175676e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875" memory_percent‚98.931884765625“}Ž@›µ‚(: ”–ãÔ•ŸÈÓ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.245' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.049000000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ìI¶‚(:¾ ”–ãÔ­°ÜÕ disk.rbd0.in‚0 disk.nvme0n1.in‚ 669.43359375) disk./.usagePercent‚71.97233918466884 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 669.43359375( proc.memory.availableMB‚ 174.96484375 proc.cpu.threads‚17% disk./.usageGB‚1072.4398651123047 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 2959.36328125 disk.dm-1.out‚ 2959.36328125$ memory_percent‚98.93209934234619# proc.memory.rssMB‚ 2857.87109375 network.recv‚ 1.182237e+06 disk.sda.out‚ 2959.36328125! disk.rbd2.out‚111.67919921875* proc.memory.percent‚17.443060874938965 disk.sda.in‚ 0.07421875 disk.rbd3.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-0.out‚48984.62890625 network.sent‚ 7.01677e+06" disk.nvme0c0n1.in‚ 669.43359375 disk.rbd0.out‚0 disk.rbd2.in‚0 cpu‚25.23771308829262 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0# disk.nvme0n1.out‚48984.62890625 disk.sda1.in‚0 disk.rbd1.out‚ 1.59619140625% disk.nvme0c0n1.out‚48984.62890625×ù‰·‚(:… £–ãÔÖàýÒ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8026 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ­ÈM¸‚(: £–ãÔŠÄ®Ö& proc.memory.availableMB‚ 174.671875% disk./.usageGB‚1072.4403038024902 disk.rbd3.in‚0 disk.sda3.out‚ 2961.06640625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.59619140625" disk.nvme0c0n1.in‚ 669.43359375 disk.drbd1777.in‚0 disk.rbd2.out‚ 111.828125 disk.sda2.in‚0( disk.nvme0c0n1.out‚48984.98583984375 disk.dm-1.out‚ 2961.06640625 disk.sda1.in‚0 disk.sda3.in‚ 0.07421875 cpu‚25.236962350539443 network.sent‚ 7.021917e+06 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚48984.98583984375 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2961.06640625 disk.sda1.out‚0) disk./.usagePercent‚71.97236862553528 disk.dm-0.in‚ 669.43359375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2857.87109375* proc.memory.percent‚17.443060874938965 proc.cpu.threads‚17 disk.drbd1777.out‚0 network.recv‚ 1.185926e+06 disk.nvme0n1.in‚ 669.43359375& disk.nvme0n1.out‚48984.98583984375$ memory_percent‚98.93388748168945™ŽÌý2¹- _wandbruntime‚3449  _runtime‚3449ôæØ2ƺÀ & lr-AdamW/pg2‚0.0002059746777672699 trainer/global_step‚3149 $ _timestamp‚1.788398385974041e+09 _runtime‚3449.533866002 _step‚254 & lr-AdamW/pg1‚0.0002059746777672699m@`˻Š& lr-AdamW/pg1‚0.0002059746777672699 & lr-AdamW/pg2‚0.0002059746777672699 trainer/global_step‚3149 $ _timestamp‚1.788398385974041e+09 _runtime‚3449.533866002 _step‚254þg"2¼- _wandbruntime‚3449  _runtime‚3449}…nÜ'½ÖO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126135046901248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22497121 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126135298489344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 297865728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚3149 _step‚255 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120393041083904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113232260275200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5724940808329 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7883983859748495e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5742023820288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83417704 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5807354238464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚118956972113681 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146000393047040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5724711838481 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83417704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59206631 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3449.535118873 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5807628164096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59206633 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ª3žIá'¾ÛO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5807354238464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚118956972113681 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146000393047040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5724711838481 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83417704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59206631 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚3449.535118873 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5807628164096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59206633 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126135046901248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22497121 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126135298489344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 297865728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚3149 _step‚255 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120393041083904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113232260275200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5724940808329 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7883983859748495e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5742023820288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83417704 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560ÿ¡ñÄ ›¿‚(: ²–ãÔè¸æÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚90.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.188200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€’XLEÀ‚(:º ²–ãÔŠÊ®Ö disk.rbd1.out‚ 1.59619140625* proc.memory.percent‚17.443060874938965 disk.sda.in‚ 0.07421875 disk.sda.out‚ 2961.8515625 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 669.43359375$ memory_percent‚98.93388748168945# proc.memory.rssMB‚ 2857.87109375% disk./.usageGB‚1072.4402923583984 disk.rbd3.out‚0 disk.rbd1.in‚0 network.sent‚ 7.028332e+06) disk./.usagePercent‚71.97236785751268 disk.drbd1777.in‚0 disk.rbd2.out‚ 111.84375 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 2961.8515625 cpu‚25.23622739611615 disk.rbd0.in‚0 disk.nvme0n1.in‚ 669.43359375% disk.nvme0n1.out‚49084.3349609375 disk.dm-0.in‚ 669.43359375" disk.dm-0.out‚49084.3349609375' disk.nvme0c0n1.out‚49084.3349609375 disk.sda1.in‚0 disk.sda3.out‚ 2961.8515625 disk.rbd0.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚ 1.190646e+06 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875& proc.memory.availableMB‚ 174.671875 disk.rbd2.in‚0ùW&2Á- _wandbruntime‚3450  _runtime‚3450ÒÒþ ÍÂàM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 276880896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _runtime‚3450.544444255 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59224812 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120430165370880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146045310880256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34735201 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34735199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22504042 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57239241 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120430183373824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126174249940992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883983869838424e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59224810 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142668617 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5726515571921 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126173998339584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113267204007936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83443805 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113267186968576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120430165370880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142642408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142668617 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5744066567168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59224812 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22504752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5809191027712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5744066567168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 OBDeviceStatsMonitor.on_train_batch_end/i™x: nactive_split.all.allocated‚57239953 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142642408 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5726744541977 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 252940800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146045587761152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚118993948549913 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120430183373824 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83417598 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 trainer/global_step‚3149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83443805 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5743832968704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83417598 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 trainer/global_step‚3149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83443805 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5743832968704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83417598 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57239241 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120430183373824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126174249940992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883983869838424e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59224810 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142668617 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5726515571921 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126173998339584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113267204007936 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83443805 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113267186968576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120430165370880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142642408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142668617 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5744066567168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59224812 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22504752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5809191027712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5744066567168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57239953 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142642408 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5726744541977 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 252940800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146045587761152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚118993948549913 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336€e„üù2Ä- _wandbruntime‚3450  _runtime‚3450\bWÅÑ /loss/FBt3/detection_loss‚3.2448600431283317 epoch‚2 _step‚257 /loss/FBt2/detection_loss‚2.8599012427859836 % _timestamp‚1.7883983869863105e+09 /loss/FRt1/assignment_loss‚6.120212030410767 2loss/SRqqt3/assignment_loss‚2.5843647924753337 1loss/SRqqt4/assignment_loss‚2.027622281242225 /loss/FBt1/detection_loss‚1.9462124635775884 /loss/FRt2/assignment_loss‚4.132538910345597 /loss/FRt3/assignment_loss‚6.163011074066162 0loss/FBt3/assignment_loss‚0.3585299522777253 /loss/FRt2/detection_loss‚2.1728954396464606 /loss/FBt4/detection_loss‚1.1743682366669983 /loss/FRt4/detection_loss‚2.0542330189184708 _runtime‚3450.546083154 0loss/FBt1/assignment_loss‚0.6218826583129461 /loss/FBt2/assignment_loss‚1.072649250427882 0loss/FBt4/assignment_loss‚0.2153648846320733 0loss/SRqqt3/detection_loss‚2.951415545665301 /loss/FRt4/assignment_loss‚2.529052622751756 2loss/SRqqt1/assignment_loss‚2.9594978014628093 1loss/SRqqt2/assignment_loss‚2.557127802177917 1loss/SRqqt2/detection_loss‚1.4202848953563114 1loss/SRqqt4/detection_loss‚1.2947202218791185 trainer/global_step‚3149 /loss/FRt1/detection_loss‚2.5213859140872956 /loss/FRt3/detection_loss‚2.4774681684818676 1loss/SRqqt1/detection_loss‚2.4821416755517323 %loss/total_loss‚57.94174092632865gëôî\ÆÖ /loss/FBt2/detection_loss‚2.8599012427859836 % _timestamp‚1.7883983869863105e+09 /loss/FRt1/assignment_loss‚6.120212030410767 2loss/SRqqt3/assignment_loss‚2.5843647924753337 1loss/SRqqt4/assignment_loss‚2.027622281242225 /loss/FBt1/detection_loss‚1.9462124635775884 /loss/FRt2/assignment_loss‚4.132538910345597 /loss/FRt3/assignment_loss‚6.163011074066162 0loss/FBt3/assignment_loss‚0.3585299522777253 /loss/FRt2/detection_loss‚2.1728954396464606 /loss/FBt4/detection_loss‚1.1743682366669983 /loss/FRt4/detection_loss‚2.0542330189184708 _runtime‚3450.546083154 0loss/FBt1/assignment_loss‚0.6218826583129461 /loss/FBt2/assignment_loss‚1.072649250427882 0loss/FBt4/assignment_loss‚0.2153648846320733 0loss/SRqqt3/detection_loss‚2.951415545665301 /loss/FRt4/assignment_loss‚2.529052622751756 2loss/SRqqt1/assignment_loss‚2.9594978014628093 1loss/SRqqt2/assignment_loss‚2.557127802177917 1loss/SRqqt2/detection_loss‚1.4202848953563114 1loss/SRqqt4/detection_loss‚1.2947202218791185 trainer/global_step‚3149 /loss/FRt1/detection_loss‚2.5213859140872956 /loss/FRt3/detection_loss‚2.4774681684818676 1loss/SRqqt1/detection_loss‚2.4821416755517323 %loss/total_loss‚57.94174092632865 /loss/FBt3/detection_loss‚3.2448600431283317 epoch‚2 _step‚257ÕFӚǂ(: Á–ãÔþ“Ó gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚99.96' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.991999999999997 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ÜIÈ‚(:¾ Á–ãÔ¶¦©Ö$ memory_percent‚98.93407821655273* proc.memory.percent‚17.443060874938965 cpu‚25.235428183382012% disk.nvme0n1.out‚49084.6865234375 disk.drbd1777.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚49084.6865234375 disk.sda.in‚ 0.07421875 network.sent‚ 7.124665e+06 disk.rbd0.in‚0 disk.sda.out‚ 2962.49609375 disk.drbd1777.out‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 174.640625) disk./.usagePercent‚71.97240830670309 disk.dm-1.out‚ 2962.49609375 disk.nvme0n1.in‚ 669.43359375 disk.sda3.out‚ 2962.49609375 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 1.59619140625 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 669.43359375' disk.nvme0c0n1.out‚49084.6865234375 disk.dm-0.in‚ 669.43359375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2857.87109375 network.recv‚ 1.196493e+06% disk./.usageGB‚1072.4408950805664 disk.sda1.out‚0 disk.rbd2.out‚ 111.96875 disk.sda3.in‚ 0.07421875!Y›É‚(: ЖãÔ‰ôßÓ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.245' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.049000000000003 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖÉ7³QÊ‚(:Æ ЖãÔüˆÖ( disk.nvme0c0n1.out‚49184.07861328125% disk./.usageGB‚1072.4407539367676 disk.nvme0n1.in‚ 669.43359375& disk.nvme0n1.out‚49184.07861328125 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.sda3.out‚ 2963.10546875 disk.rbd0.in‚0 network.recv‚ 1.202887e+06# disk.dm-0.out‚49184.07861328125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.59619140625 disk.rbd0.out‚0$ memory_percent‚98.93414974212646 cpu‚25.234638629754517 proc.cpu.threads‚17* proc.memory.percent‚17.443060874938965 network.sent‚ 7.222745e+06 disk.dm-0.in‚ 669.43359375 disk.sda.out‚ 2963.10546875 disk.drbd1777.out‚0 disk.rbd2.out‚ 113.49609375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 669.43359375) disk./.usagePercent‚71.97239883442433 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 2963.10546875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875( proc.memory.availableMB‚ 174.62890625# proc.memory.rssMB‚ 2857.87109375PIt¯Ë‚(:„ ß–ãÔ飂Ó gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.715 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½¶ÛÚPÌ‚(:Å ß–ãÔ®ä¾Ö disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚98.93412590026855 disk.sda.out‚ 2964.18359375 disk.rbd1.out‚ 1.59619140625( disk.nvme0c0n1.out‚49184.40283203125 disk.dm-0.in‚ 669.43359375 disk.sda1.in‚0 cpu‚25.233916995422703 disk.rbd2.in‚0' proc.memory.availableMB‚ 174.6328125# proc.memory.rssMB‚ 2857.87109375 disk.drbd1777.out‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.sda.in‚ 0.07421875 network.sent‚ 7.227949e+06 network.recv‚ 1.206748e+06 disk.rbd3.in‚0 disk.sda3.out‚ 2964.18359375 disk.nvme0n1.in‚ 669.43359375& disk.nvme0n1.out‚49184.40283203125* proc.memory.percent‚17.443060874938965) disk./.usagePercent‚71.97240651465034 disk.rbd1.in‚0 disk.dm-1.out‚ 2964.18359375 proc.cpu.threads‚17% disk./.usageGB‚1072.4408683776855 disk.rbd2.out‚ 113.55859375" disk.nvme0c0n1.in‚ 669.43359375# disk.dm-0.out‚49184.40283203125/°#2Í- _wandbruntime‚3501  _runtime‚3501ì_·kÇÎÁ & lr-AdamW/pg1‚0.0002051096304999708 & lr-AdamW/pg2‚0.0002051096304999708 trainer/global_step‚3199 % _timestamp‚1.7883984381087515e+09 _runtime‚3501.668629053 _step‚258§®ÌÏÆ % _timestamp‚1.7883984381087515e+09 _runtime‚3501.668629053 _step‚258 & lr-AdamW/pg1‚0.0002051096304999708 & lr-AdamW/pg2‚0.0002051096304999708 trainer/global_step‚3199‚ИnÅ2Ð- _wandbruntime‚3501  _runtime‚3501ÿý©[Û'ÑÕO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122250137576960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35257198 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5832495654912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚742 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128082884819968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114978594944000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60115583 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5832495654912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122250155579904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58097534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚120793753054897 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128082633231872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5832729240064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128082633231872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84724806 trainer/global_step‚3199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148260463768064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788398438109607e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22840338 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60115583 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84698625 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22841078 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5899042877440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114978611983360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Ug` ÒÚO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60115583 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84698625 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22841078 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5899042877440 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114978611983360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122250137576960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35257198 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5832495654912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚742 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128082884819968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114978594944000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60115583 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5832495654912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122250155579904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58097534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚120793753054897 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128082633231872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5832729240064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128082633231872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84724806 trainer/global_step‚3199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148260463768064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788398438109607e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22840338 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307ƒÔóW¹2Ó- _wandbruntime‚3502  _runtime‚3502j Yå&ÔßM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60133762 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚721 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142405065337344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5816715835129 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144858460 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58115158 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142405089277440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128121584670208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58115879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148305671645696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5900865540608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84750907 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84724700 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5834538401792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60133760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22847309 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84750907 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128121836271616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5816944805185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 $ _timestamp‚1.788398439074898e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 _step‚260 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122287297869824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22848028 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120830483481921 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5834304803328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144884669 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5834304803328 trainer/global_step‚3199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120830237472505 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60133760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128121836271616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122287279866880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84724700 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144858460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5900606308352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122287279866880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144884669 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35267851 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115013538676736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115013521637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60133762 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144884669 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5834304803328 trainer/global_step‚3199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120830237472505 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60133760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128121836271616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122287279866880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84724700 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144858460 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5900606308352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122287279866880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144884669 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35267851 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115013538676736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115013521637376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60133762 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60133762 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚721 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142405065337344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5816715835129 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144858460 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58115158 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142405089277440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128121584670208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58115879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148305671645696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5900865540608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0„Hjüf2Ö- _wandbruntime‚3502  _runtime‚3502N»L×Æ 0loss/FBt2/assignment_loss‚0.4674309328771554 .loss/FBt3/assignment_loss‚0.89851165041952 /loss/FRt2/detection_loss‚2.0888167416055996 -loss/FRt3/detection_loss‚2.23589145409919 /loss/FBt3/detection_loss‚3.0589916825294496 _runtime‚3502.636711764 .loss/FRt2/assignment_loss‚3.29468785127004 0loss/FRt3/assignment_loss‚5.8349774451482865 .loss/FBt1/detection_loss‚3.757444308354555 %loss/total_loss‚57.93681356271564 1loss/SRqqt1/assignment_loss‚2.673507160610623 -loss/FRt1/detection_loss‚2.55001757144928 .loss/FBt4/detection_loss‚1.197087166392628 1loss/SRqqt3/assignment_loss‚2.843012796714902 epoch‚2 % _timestamp‚1.7883984390769916e+09 /loss/FRt4/detection_loss‚2.0966615308733547 /loss/SRqqt1/detection_loss‚2.45245494445165 /loss/FRt1/assignment_loss‚5.945123519897461 0loss/FRt4/assignment_loss‚2.8145864161790586 1loss/SRqqt2/detection_loss‚1.3233930797953355 /loss/SRqqt3/detection_loss‚2.45178116299212 0loss/FBt1/assignment_loss‚0.5610062746384826 0loss/FBt4/assignment_loss‚0.3377361517965215 1loss/SRqqt4/detection_loss‚1.4497474125276009 trainer/global_step‚3199 _step‚261 1loss/SRqqt4/assignment_loss‚2.124238229046265 .loss/FBt2/detection_loss‚3.246124872794518 2loss/SRqqt2/assignment_loss‚2.2335832062520478Éí/€QØË 1loss/SRqqt3/assignment_loss‚2.843012796714902 epoch‚2 % _timestamp‚1.7883984390769916e+09 /loss/FRt4/detection_loss‚2.0966615308733547 /loss/SRqqt1/detection_loss‚2.45245494445165 /loss/FRt1/assignment_loss‚5.945123519897461 0loss/FRt4/assignment_loss‚2.8145864161790586 1loss/SRqqt2/detection_loss‚1.3233930797953355 /loss/SRqqt3/detection_loss‚2.45178116299212 0loss/FBt1/assignment_loss‚0.5610062746384826 0loss/FBt4/assignment_loss‚0.3377361517965215 1loss/SRqqt4/detection_loss‚1.4497474125276009 trainer/global_step‚3199 _step‚261 1loss/SRqqt4/assignment_loss‚2.124238229046265 .loss/FBt2/detection_loss‚3.246124872794518 2loss/SRqqt2/assignment_loss‚2.2335832062520478 0loss/FBt2/assignment_loss‚0.4674309328771554 .loss/FBt3/assignment_loss‚0.89851165041952 /loss/FRt2/detection_loss‚2.0888167416055996 -loss/FRt3/detection_loss‚2.23589145409919 /loss/FBt3/detection_loss‚3.0589916825294496 _runtime‚3502.636711764 .loss/FRt2/assignment_loss‚3.29468785127004 0loss/FRt3/assignment_loss‚5.8349774451482865 .loss/FBt1/detection_loss‚3.757444308354555 %loss/total_loss‚57.93681356271564 1loss/SRqqt1/assignment_loss‚2.673507160610623 -loss/FRt1/detection_loss‚2.55001757144928 .loss/FBt4/detection_loss‚1.197087166392628…8IŽÙ‚(:ƒ î–ãÔïó‡Ó gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚96.02' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ñ±L£Ú‚(:à î–ãÔí­öÕ% disk./.usageGB‚1072.4407577514648 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.59619140625( disk.nvme0c0n1.out‚49184.92138671875 network.recv‚ 1.211303e+06 disk.rbd0.out‚0& disk.nvme0n1.out‚49184.92138671875 disk.dm-0.in‚ 669.43359375# disk.dm-0.out‚49184.92138671875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 2985.5234375 disk.rbd2.in‚0 disk.rbd2.out‚ 113.57421875# proc.memory.rssMB‚ 2857.87109375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ memory_percent‚98.93438816070557 network.sent‚ 7.234274e+06) disk./.usagePercent‚71.97239909043186 disk.sda.out‚ 2985.5234375 disk.drbd1777.out‚0 cpu‚25.233145146118616" disk.nvme0c0n1.in‚ 669.43359375( proc.memory.availableMB‚ 174.58984375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 669.43359375 disk.sda1.out‚0 06N« disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 2985.5234375* proc.memory.percent‚17.443060874938965 disk.rbd0.in‚0 disk.dm-1.in‚ 0.07421875Ö°P†Û‚(:… ý–ãÔô½æÓ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚94.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9418 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/]uóCÜ‚(:¸ ý–ãÔÔ¨ÈÕ disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 669.43359375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 669.43359375 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 1.68212890625 disk.dm-0.out‚ 49284.1640625# proc.memory.rssMB‚ 2857.87109375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 669.43359375* proc.memory.percent‚17.443060874938965 cpu‚25.2323770584401% disk./.usageGB‚1072.4408721923828 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 49284.1640625 disk.sda1.out‚0 disk.sda3.out‚ 3001.01953125 disk.sda.in‚ 0.07421875 disk.sda.out‚ 3001.01953125 network.recv‚ 1.217108e+06 disk.rbd2.out‚ 113.57421875 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 3001.01953125" disk.nvme0n1.out‚ 49284.1640625 disk.drbd1777.in‚0$ memory_percent‚98.93457889556885 network.sent‚ 7.341898e+06 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 174.55859375) disk./.usagePercent‚71.97240677065788UHµ7›Ý‚(: Œ—ãÔÿîåÓ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚93.604' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.720799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 “tíSÞ‚(:È Œ—ãÔš–íÕ disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 3002.24609375# disk.dm-0.out‚49284.97314453125 disk.sda.in‚ 0.07421875 disk.sda1.out‚0* proc.memory.percent‚17.443060874938965 network.sent‚ 7.348351e+06 disk.sda2.in‚0 disk.dm-1.out‚ 3002.24609375 disk.nvme0n1.in‚ 669.43359375 disk.dm-0.in‚ 669.43359375 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 1.68212890625' proc.memory.availableMB‚ 174.5546875 cpu‚25.231675870634852 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875$ memory_percent‚98.93460273742676# proc.memory.rssMB‚ 2857.87109375) disk./.usagePercent‚71.97240062647707% disk./.usageGB‚1072.4407806396484 disk.sda1.in‚0 network.recv‚ 1.221744e+06! disk.rbd2.out‚113.67626953125" disk.nvme0c0n1.in‚ 669.43359375( disk.nvme0c0n1.out‚49284.97314453125 disk.dm-1.in‚ 0.07421875& disk.nvme0n1.out‚49284.97314453125 disk.sda.out‚ 3002.24609375 disk.drbd1777.out‚0õPwª2ß- _wandbruntime‚3553  _runtime‚3553ÒKhÇàÁ trainer/global_step‚3249 % _timestamp‚1.7883984902530951e+09 _runtime‚3553.812949997 _step‚262 & lr-AdamW/pg1‚0.0002042250765894325 & lr-AdamW/pg2‚0.0002042250765894325“ø‚ÌáÆ _runtime‚3553.812949997 _step‚262 & lr-AdamW/pg1‚0.0002042250765894325 & lr-AdamW/pg2‚0.0002042250765894325 trainer/global_step‚3249 % _timestamp‚1.7883984902530951e+09†±ÆŠ¦2â- _wandbruntime‚3553  _runtime‚3553@èÇaÜ'ãÖO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61024533 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86031908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122630287986905 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130030471150592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124107252072960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147030258 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5922967489536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124107270075904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86005727 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5923201074688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61024531 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35789846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86031908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130030219562496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150521131312128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130030219562496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61024533 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58974911 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚116724929612800 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5990205728256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58974182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124107270075904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144530631912448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86031908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122630287986905 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130030471150592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124107252072960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147030258 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5922967489536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124107270075904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86005727 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5923201074688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61024531 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35789846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86031908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130030219562496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150521131312128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130030219562496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61024533 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58974911 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚116724929612800 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5990205728256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58974182 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960‡z4ÀÃ2å- _wandbruntime‚3554  _runtime‚3554Yȼ)S!æàM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5925010236416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 _step‚264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5925010236416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5907145068393 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58992432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5992294508032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61042712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35800501 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122667018413929 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚124144412365824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130069171000832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5924776637952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚747 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144574034762240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150566046097920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚116759856306176 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86031802 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61042710 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61042712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144574010822144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122666772404513 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5924776637952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/rKï#¶“eserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883984912672124e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124144394362880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚130069422602240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147100721 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58991685 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147074512 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 trainer/global_step‚3249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23191186 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253_B°üë&çåM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚3554.827933835 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147074512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35800499 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61042710 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61042712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144574010822144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122666772404513 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5924776637952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 283172352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883984912672124e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124144394362880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚130069422602240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147100721 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58991685 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147074512 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 trainer/global_step‚3249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23191186 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5925010236416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 _step‚264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5925010236416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5907145068393 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58992432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5992294508032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 259232256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61042712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35800501 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122667018413929 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚124144412365824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130069171000832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5924776637952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚747 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144574034762240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150566046097920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚116759856306176 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86031802 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PYOî‚(:Ä ª—ãÔÅžÎÖ' proc.memory.availableMB‚ 174.5234375$ disk./.usageGB‚1072.440803527832 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚49385.25537109375 disk.sda.out‚ 3019.3515625 proc.cpu.threads‚17 network.recv‚ 1.231412e+06 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.74462890625 disk.dm-1.in‚ 0.07421875* proc.memory.percent‚17.443060874938965 disk.dm-0.in‚ 670.05078125 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚98.93479347229004# proc.memory.rssMB‚ 2857.87109375 disk.nvme0n1.in‚ 670.05078125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.sda.in‚ 0.07421875( disk.nvme0c0n1.out‚49385.25537109375 disk.dm-1.out‚ 3019.3515625 cpu‚25.230149443619066 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 3019.3515625" disk.nvme0c0n1.in‚ 670.05078125 network.sent‚ 7.461957e+06& disk.nvme0n1.out‚49385.25537109375! disk.rbd2.out‚115.32470703125 disk.sda2.out‚0) disk./.usagePercent‚71.97240216252226 disk.drbd1777.in‚0~ j÷œï‚(:‘ ¹—ãÔª¥ºÔ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.262' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.252399999999998 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oy5ëSð‚(:È ¹—ãÔ£âÖ( disk.nvme0c0n1.out‚49505.06689453125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 670.05078125 disk.sda.in‚ 0.07421875# proc.memory.rssMB‚ 2857.87109375* proc.memory.percent‚17.443060874938965 network.recv‚ 1.242731e+06 disk.sda3.in‚ 0.07421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 3033.19921875 disk.rbd0.in‚0& disk.nvme0n1.out‚49505.12939453125# disk.dm-0.out‚49505.19189453125# memory_percent‚98.9332914352417 network.sent‚ 7.470511e+06 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 174.76953125 disk.sda2.out‚0 disk.rbd1.out‚ 1.77685546875" disk.nvme0c0n1.in‚ 670.05078125 disk.nvme0n1.in‚ 670.05078125 disk.sda.out‚ 3033.19921875 proc.cpu.threads‚17) disk./.usagePercent‚71.93804083127654% disk./.usageGB‚1071.9287948608398! disk.rbd2.out‚115.32470703125 disk.dm-1.out‚ 3033.19921875 cpu‚25.229398914797244˜5ãÚ›ñ‚(: È—ãÔó˜îÒ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.51' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.701999999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Ÿ$Gò‚(:¼ È—ãÔÓÑÕ! disk.rbd2.out‚115.40283203125 disk.sda3.out‚ 3040.36328125( proc.memory.availableMB‚ 174.51953125 cpu‚25.22867328674436$ disk.nvme0n1.out‚49583.103515625 disk.dm-0.in‚ 670.05078125 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 3040.36328125 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.77685546875$ memory_percent‚98.93481731414795* proc.memory.percent‚17.443084716796875 network.sent‚ 7.476714e+06 disk.nvme0n1.in‚ 670.05078125 disk.drbd1777.in‚0 network.recv‚ 1.247483e+06) disk./.usagePercent‚71.39133034171805" disk.nvme0c0n1.in‚ 670.05078125 disk.dm-1.out‚ 3040.36328125 disk.rbd0.out‚0$ disk./.usageGB‚1063.782413482666 disk.dm-1.in‚ 0.07421875! disk.dm-0.out‚49583.103515625 disk.rbd1.in‚0 proc.memory.rssMB‚2857.875& disk.nvme0c0n1.out‚49583.103515625 disk.sda.in‚ 0.07421875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda3.in‚ 0.07421875v^gh2ó- _wandbruntime‚3602  _runtime‚3602Ë‹´'ÆôÀ & lr-AdamW/pg2‚0.0002033212003753642 trainer/global_step‚3299 $ _timestamp‚1.788398539068349e+09 _runtime‚3602.628138429 _step‚266 & lr-AdamW/pg1‚0.0002033212003753642)%:=ËõÅ $ _timestamp‚1.788398539068349e+09 _runtime‚3602.628138429 _step‚266 & lr-AdamW/pg1‚0.0002033212003753642 & lr-AdamW/pg2‚0.0002033212003753642 trainer/global_step‚3299ŠêÇÌ·2ö- _wandbruntime‚3602  _runtime‚3602f±]#Û'÷ÕO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 285282816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118471281320960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152781493924352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152781208641536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5995541597953 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 trainer/global_step‚3299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149272493 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61933481 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87339010 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6081892587008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 261342720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 $ _timestamp‚1.788398539069076e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131978057481216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131977805893120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124466576909705 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59855701 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125964366568960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125964366568960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36322496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5995312628105 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960{¢@@ÃøÚO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131977805893120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87312829 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6013672909312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allªƒ !ocated‚61933483 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23532494 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146699577397248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61933483 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146699601337344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚267 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87312829 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149246310 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23533203 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131977805893120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124466576909705 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59855701 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125964366568960 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125964366568960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36322496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5995312628105 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 285282816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118471281320960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152781493924352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152781208641536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5995541597953 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 trainer/global_step‚3299 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149272493 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61933481 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87339010 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6081892587008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 261342720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 $ _timestamp‚1.788398539069076e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131978057481216 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59873020 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149290564 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152826707430912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149316773 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61951660 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6015248472576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149290564 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59872284 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 trainer/global_step‚3299 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61951662 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87338904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124503307336521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 _runtime‚3603.528618608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61951662 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6083727183872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6083467951616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124503553345937 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126001526861824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6083727183872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6083467951616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124503553345937 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126001526861824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59873020 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149290564 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152826707430912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149316773 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61951660 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6015248472576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149290564 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59872284 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 trainer/global_step‚3299 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61951662 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87338904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124503307336521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845ŒCùŠÁ2ü- _wandbruntime‚3603  _runtime‚3603+žVÌSýÍ 0loss/FBt1/assignment_loss‚0.5655016881808277 0loss/FBt3/assignment_loss‚1.1012016013264656 .loss/FRt1/detection_loss‚2.491404120411192 % _timestamp‚1.7883985399700365e+09 2loss/SRqqt4/assignment_loss‚2.3613733985594343 0loss/SRqqt3/detection_loss‚2.721400339404742 1loss/SRqqt4/detection_loss‚1.0506952520992074 -loss/FBt1/detection_loss‚1.42480635387754 /loss/FBt4/detection_loss‚1.1600387606656912 trainer/global_step‚3299 /loss/FRt3/assignment_loss‚5.893609313403859 0loss/SRqqt1/assignment_loss‚2.53556117967323 1loss/FBt4/assignment_loss‚0.24275627320507323 %loss/total_loss‚57.65228974974555 epoch‚2 _runtime‚3603.529747195 0loss/SRqqt1/detection_loss‚2.944769309626685 _step‚269 0loss/FRt1/assignment_loss‚5.7579255785260886 1loss/SRqqt2/assignment_loss‚2.286051371630202 .loss/FRt3/detection_loss‚2.545689551817144 .loss/FRt4/detection_loss‚1.713695403886959 .loss/FBt2/detection_loss‚5.098314166069031 0loss/FRt4/assignment_loss‚2.4095083428546786 1loss/SRqqt2/detection_loss‚1.2598942635541266 /loss/FRt2/assignment_loss‚4.531324301447187 2loss/SRqqt3/assignment_loss‚2.6487188855806987 1loss/FBt2/assignment_loss‚0.21549574006348848 .loss/FBt3/detection_loss‚2.487910976012548 /loss/FRt2/detection_loss‚2.2046435778694495Òœ@’XþÒ epoch‚2 _runtime‚3603.529747195 0loss/SRqqt1/detection_loss‚2.944769309626685 _step‚269 0loss/FRt1/assignment_loss‚5.7579255785260886 1loss/SRqqt2/assignment_loss‚2.286051371630202 .loss/FRt3/detection_loss‚2.545689551817144 .loss/FRt4/detection_loss‚1.713695403886959 .loss/FBt2/detection_loss‚5.098314166069031 0loss/FRt4/assignment_loss‚2.4095083428546786 1loss/SRqqt2/detection_loss‚1.2598942635541266 /loss/FRt2/assignment_loss‚4.531324301447187 2loss/SRqqt3/assignment_loss‚2.6487188855806987 1loss/FBt2/assignment_loss‚0.21549574006348848 .loss/FBt3/detection_loss‚2.487910976012548 /loss/FRt2/detection_loss‚2.2046435778694495 0loss/FBt1/assignment_loss‚0.5655016881808277 0loss/FBt3/assignment_loss‚1.1012016013264656 .loss/FRt1/detection_loss‚2.491404120411192 % _timestamp‚1.7883985399700365e+09 2loss/SRqqt4/assignment_loss‚2.3613733985594343 0loss/SRqqt3/detection_loss‚2.721400339404742 1loss/SRqqt4/detection_loss‚1.0506952520992074 -loss/FBt1/detection_loss‚1.42480635387754 /loss/FBt4/detection_loss‚1.1600387606656912 trainer/global_step‚3299 /loss/FRt3/assignment_loss‚5.893609313403859 0loss/SRqqt1/assignment_loss‚2.53556117967323 1loss/FBt4/assignment_loss‚0.24275627320507323 %loss/total_loss‚57.65228974974555–êûÿ‚(:„ ×—ãÔæăÓ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚89.405' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.881 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©¸Ê5€‚(:° ×—ãÔ£ÉÆÕ disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.05078125 disk.sda.out‚ 3062.93359375) disk./.usagePercent‚71.25121434613791 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 670.05078125! proc.memory.availableMB‚174.5 cpu‚25.227663847073984 disk.sda3.out‚ 3062.93359375 disk.dm-0.in‚ 670.05078125 disk.sda.in‚ 0.07421875 disk.sda2.in‚0 disk.dm-1.out‚ 3062.93359375 disk.dm-0.out‚49629.30859375 proc.memory.rssMB‚2857.875* proc.memory.percent‚17.443084716796875 proc.cpu.threads‚17# memory_percent‚98.9349365234375 network.sent‚ 7.481895e+06 network.recv‚ 1.251207e+06% disk.nvme0c0n1.out‚49629.30859375% disk./.usageGB‚1061.6945838928223 disk.sda1.out‚0! disk.rbd2.out‚115.40283203125 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.83984375# disk.nvme0n1.out‚49629.30859375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd1.¹yŠin‚0ø¡Ü‚(:„ æ—ãÔûê¡Ó gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.054 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 VóH‚‚(:½ æ—ãÔêƒÕ disk.rbd2.out‚115.4892578125& proc.memory.availableMB‚ 174.484375 disk.rbd2.in‚0 disk.rbd3.in‚0 network.sent‚ 7.591727e+06) disk./.usagePercent‚70.79270843641903( disk.nvme0c0n1.out‚49752.72119140625 disk.dm-1.in‚ 0.07421875$ memory_percent‚98.93503189086914 disk.dm-1.out‚ 3064.71484375 disk.sda1.in‚0 cpu‚25.22687316484525% disk./.usageGB‚1054.8625144958496 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 disk.sda.out‚ 3064.71484375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.07421875# disk.dm-0.out‚49752.72119140625 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.443084716796875 network.recv‚ 1.259246e+06 disk.sda3.out‚ 3064.71484375& disk.nvme0n1.out‚49752.72119140625 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.83984375" disk.nvme0c0n1.in‚ 670.05078125 disk.rbd0.in‚0 disk.dm-0.in‚ 670.05078125 disk.sda1.out‚0 proc.memory.rssMB‚2857.875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 670.05078125û‰z‘ƒ‚(:† õ—ãÔù‹ˆÓ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.739' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5478 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 YJ„‚(:¿ õ—ãÔÔ¼îÕ( disk.nvme0c0n1.out‚49919.16259765625 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 7.601032e+06 disk.dm-1.in‚ 0.07421875$ disk./.usageGB‚1054.060718536377 disk.nvme0n1.in‚ 670.05078125 disk.sda.out‚ 3066.47265625 disk.rbd1.out‚ 1.90283203125 cpu‚25.22600345023985 disk.sda.in‚ 0.07421875 disk.sda2.in‚0* proc.memory.percent‚17.443084716796875" disk.nvme0c0n1.in‚ 670.05078125 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875# memory_percent‚98.9351749420166) disk./.usagePercent‚70.73889923682721 disk.dm-1.out‚ 3066.47265625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 3066.47265625 proc.memory.rssMB‚2857.875 network.recv‚ 1.270398e+06& disk.nvme0n1.out‚49919.16259765625 disk.dm-0.in‚ 670.05078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚117.0791015625 disk.rbd0.in‚0# disk.dm-0.out‚49919.16259765625 disk.drbd1777.in‚0' proc.memory.availableMB‚ 174.4609375 proc.cpu.threads‚17»K ä2…- _wandbruntime‚3650  _runtime‚3650bŽjÊȆ ' lr-AdamW/pg1‚0.00020239819022421416 ' lr-AdamW/pg2‚0.00020239819022421416 trainer/global_step‚3349 $ _timestamp‚1.788398586475462e+09 _runtime‚3650.035318162 _step‚270c*?/Í‡Ç ' lr-AdamW/pg1‚0.00020239819022421416 ' lr-AdamW/pg2‚0.00020239819022421416 trainer/global_step‚3349 $ _timestamp‚1.788398586475462e+09 _runtime‚3650.035318162 _step‚270Ž9¿Eš2ˆ- _wandbruntime‚3650  _runtime‚3650àÞx Ü'‰ÖO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151462362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127821481064960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 % _timestamp‚1.7883985864763954e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148868546822144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚126303111841713 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6085741861161 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133925392223744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60730177 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6085512891313 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3650.036474481 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151488545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120217615989760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127821481064960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62842431 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6173300324864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62842433 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6104144743936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155041847147008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281088512 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133925392223744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88646112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36855146 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126303357850921 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6173043176448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚271 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88619931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23874327 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120217598950400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127821499067904 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151462362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127821481064960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 % _timestamp‚1.7883985864763954e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148868546822144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚126303111841713 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6085741861161 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133925392223744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60730177 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6085512891313 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3650.036474481 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151488545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120217615989760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127821481064960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62842431 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6173300324864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62842433 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6104144743936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155041847147008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 281088512 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133925392223744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88646112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36855146 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126303357850921 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6173043176448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚271 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88619931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23874327 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120217598950400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253MÈÜÐ2‹- _wandbruntime‚3650  _runtime‚3650þ¶Ö"ŒàM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6105720307200 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88672213 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23881409 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚126339842268529 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127858623354880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6105953905664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3349 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60747929 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6105720307200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23882128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155087078866432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148911925731840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263426560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60747208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126340088277945 _runtime‚3650.941188197 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133964595263488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36865799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127858641357824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6087316624753 _step‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6105953905664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133964595263488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62860610 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6175153134592 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6087545594809 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88646006 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148911901791744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120252542683136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7883985873807225e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151506616 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151532825 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127858641357824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120252525643776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155086791499776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88646006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 MCDeviceStatsMonitor.on_train_batcß?±Báh_end/allocation.small_pool.current‚26207 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287366656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6174889708032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151532825 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151506616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62860612 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62860610 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88672213 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127858623354880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62860612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 :bë&åM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62860610 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6175153134592 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6087545594809 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88646006 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148911901791744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120252542683136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7883985873807225e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151506616 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151532825 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127858641357824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120252525643776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155086791499776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88646006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287366656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6174889708032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151532825 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151506616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62860612 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62860610 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88672213 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127858623354880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62860612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6105720307200 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88672213 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23881409 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚126339842268529 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127858623354880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6105953905664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3349 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60747929 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6105720307200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23882128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155087078866432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148911925731840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263426560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60747208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126340088277945 _runtime‚3650.941188197 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133964595263488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36865799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127858641357824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6087316624753 _step‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6105953905664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133964595263488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2ÑQw2Ž- _wandbruntime‚3650  _runtime‚36505]XÒ /loss/FRt1/assignment_loss‚6.096505507826805 0loss/FRt4/assignment_loss‚3.0105835910026846 2loss/SRqqt2/assignment_loss‚2.4892810997439594 0loss/FBt1/assignment_loss‚0.5252635206348454 0loss/SRqqt3/detection_loss‚2.563769580796361 2loss/SRqqt3/assignment_loss‚2.3844189578667283 1loss/FBt2/assignment_loss‚0.19822306763380765 %loss/total_loss‚57.25692267282758 epoch‚2 _runtime‚3650.942630906 0loss/FBt3/assignment_loss‚0.5081670177540738 /loss/FRt4/detection_loss‚1.8911279694965253 1loss/SRqqt4/detection_loss‚1.0590697407722474 .loss/FBt2/detection_loss‚3.685505437850952 .loss/FBt3/detection_loss‚3.575312554836273 $ _timestamp‚1.788398587382938e+09 /loss/FRt1/detection_loss‚2.3409679420292377 /loss/FRt3/detection_loss‚2.8358603533771305 /loss/FRt2/detection_loss‚2.0668345373123884 1loss/SRqqt1/detection_loss‚2.9066870331764223 _step‚273 /loss/FRt2/assignment_loss‚4.297373863868415 /loss/FRt3/assignment_loss‚5.696593284606934 1loss/SRqqt1/assignment_loss‚2.925727146863937 1loss/SRqqt4/assignment_loss‚2.174034403837644 1loss/FBt4/assignment_loss‚0.24780410302614655 1loss/SRqqt2/detection_loss‚1.4711830472437346 /loss/FBt1/detection_loss‚0.9985388102024904 /loss/FBt4/detection_loss‚1.3080901010678365 trainer/global_step‚3349L³gÌ]× %loss/total_loss‚57.25692267282758 epoch‚2 _runtime‚3650.942630906 0loss/FBt3/assignment_loss‚0.5081670177540738 /loss/FRt4/detection_loss‚1.8911279694965253 1loss/SRqqt4/detection_loss‚1.0590697407722474 .loss/FBt2/detection_loss‚3.685505437850952 .loss/FBt3/detection_loss‚3.575312554836273 $ _timestamp‚1.788398587382938e+09 /loss/FRt1/detection_loss‚2.3409679420292377 /loss/FRt3/detection_loss‚2.8358603533771305 /loss/FRt2/detection_loss‚2.0668345373123884 1loss/SRqqt1/detection_loss‚2.9066870331764223 _step‚273 /loss/FRt2/assignment_loss‚4.297373863868415 /loss/FRt3/assignment_loss‚5.696593284606934 1loss/SRqqt1/assignment_loss‚2.925727146863937 1loss/SRqqt4/assignment_loss‚2.174034403837644 1loss/FBt4/assignment_loss‚0.24780410302614655 1loss/SRqqt2/detection_loss‚1.4711830472437346 /loss/FBt1/detection_loss‚0.9985388102024904 /loss/FBt4/detection_loss‚1.3080901010678365 trainer/global_step‚3349 /loss/FRt1/assignment_loss‚6.096505507826805 0loss/FRt4/assignment_loss‚3.0105835910026846 2loss/SRqqt2/assignment_loss‚2.4892810997439594 0loss/FBt1/assignment_loss‚0.5252635206348454 0loss/SRqqt3/detection_loss‚2.563769580796361 2loss/SRqqt3/assignment_loss‚2.3844189578667283 1loss/FBt2/assignment_loss‚0.19822306763380765‘pe§š‘‚(: „˜ãÔ“ŠÓ gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.58' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.715999999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍE’‚(:º „˜ãÔ­ÇïÖ disk.rbd2.in‚0* proc.memory.percent‚17.443084716796875% disk./.usageGB‚1054.0613250732422 disk.sda.in‚ 0.07421875 disk.sda3.out‚ 3067.04296875 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 7.60714e+06 disk.dm-1.in‚ 0.07421875 disk.rbd1.out‚ 1.90283203125# memory_percent‚98.9349365234375 proc.memory.rssMB‚2857.875 cpu‚25.225273114986322 disk.rbd3.in‚0! proc.memory.availableMB‚174.5) disk./.usagePercent‚70.73893994202514 disk.nvme0n1.in‚ 670.05078125 disk.dm-0.in‚ 670.05078125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚49919.82177734375 disk.sda3.in‚ 0.07421875 network.recv‚ 1.274306e+06 disk.dm-1.out‚ 3067.04296875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚117.2041015625 disk.rbd3.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚49919.82177734375 disk.sda.out‚ 3067.04296875" disk.nvme0c0n1.in‚ 670.05078125( disk.nvme0c0n1.out‚49919.82177734375Lj®‹“‚(:… “˜ãÔ¤Ü€Ó gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3348 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢Ë‹K”‚(:À “˜ãÔ£ÂØ network.sent‚ 7.714527e+06% disk./.usageGB‚1054.0614776611328 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚117.2197265625& proc.memory.availableMB‚ 174.984375 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.90283203125 cpu‚25.22454528548284 disk.sda2.out‚0 disk.dm-1.out‚ 3067.57421875& disk.nvme0n1.out‚49920.69775390625 disk.dm-0.in‚ 670.05078125# disk.dm-0.out‚49920.69775390625" disk.nvme0c0n1.in‚ 670.05078125 disk.rbd0.in‚0$ memory_percent‚98.93198013305664) disk./.usagePercent‚70.73895018232652 disk.sda2.in‚0 disk.sda3.out‚ 3067.57421875 disk.sda.out‚ 3067.57421875 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0( disk.nvme0c0n1.out‚49920.69775390625 disk.sda.in‚ 0.07421875* proc.memory.percent‚17.443084716796875 network.recv‚ 1.279442e+06 disk.sda1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 670.05078125 proc.memory.rssMB‚2857.875ˆòß‘•‚(:† ¢˜ãÔúáÿÒ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.317' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6634 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åõ!K–‚(:À ¢˜ãÔ’í°Õ disk.sda3.out‚ 3068.09765625 disk.rbd1.in‚0( disk.nvme0c0n1.out‚49922.72998046875# disk.dm-0.out‚49922.72998046875 proc.cpu.threads‚17 disk.drbd1777.in‚0! disk.rbd2.out‚117.36865234375 disk.rbd3.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-0.in‚ 670.1484375 disk.sda.in‚ 0.07421875( proc.memory.availableMB‚ 173.62109375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚2857.875 disk.drbd1777.out‚0 disk.dm-1.out‚ 3068.09765625& disk.nvme0n1.out‚49922.72998046875 disk.sda1.out‚0* proc.memory.percent‚17.443084716796875 disk.sda.out‚ 3068.09765625 network.sent‚ 7.731383e+06 network.recv‚ 1.304931e+06 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.out‚0 cpu‚25.22381199023747) disk./.usagePercent‚70.73894531818337 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 670.1484375 disk.nvme0n1.in‚ 670.1484375$ memory_percent‚98.94030094146729% disk./.usageGB‚1054.0614051818848 disk.rbd1.out‚ 1.90283203125P¾é]2—- _wandbruntime‚3697  _runtime‚3697GØRȘ trainer/global_step‚3399 $ _timestamp‚1.788398633580299e+09 _runtime‚3697.140142073 _step‚274 ' lr-AdamW/pg1‚0.00020145623848991437 ' lr-AdamW/pg2‚0.00020145623848991437ż‹Í™Ç ' lr-AdamW/pg1‚0.00020145623848991437 ' lr-AdamW/pg2‚0.00020145623848991437 trainer/global_step‚3399 $ _timestamp‚1.788398633580299e+09 _runtime‚3697.140142073 _step‚274’ÒJ¥#2š- _wandbruntime‚3697  _runtime‚3697V^áuÜ'›ÖO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157302223264256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157301931689984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61609002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 _step‚275 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6194616578560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6194382993408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63751381 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚128139892782929 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151037468366848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6264463323136 trainer/global_step‚3399 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89953214 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129678595560960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89927033 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121963950658560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89927033 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135872978554368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24221206 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121963933619200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129678613563904 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6194616578560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _runtime‚3697.141139613 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129678613563904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚128139646773721 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89953214 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267634176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63751383 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151037492306944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6175942124369 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37387798 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6264730957312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153704597 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24221958 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61609756 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6194382993408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135872978554368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135873230142464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89953214 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267634176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63751383 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151037492306944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6175942124369 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37387798 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6264730957312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153704597 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24221958 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61609756 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6194382993408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135872978554368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135873230142464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89953214 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129678595560960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89927033 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121963950658560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89927033 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135872978554368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24221206 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121963933619200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129678613563904 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6194616578560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _runtime‚3697.141139613 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129678613563904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚128139646773721 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504“™æïÕ2- _wandbruntime‚3698  _runtime‚3698¶Vìõæ&žàM % _timestamp‚1.7883986344791481e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6266286603776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6177745858017 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63769560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6196192141824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153722668 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128176623209953 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129715755853824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚737 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121998860312576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63769562 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61627199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63769560 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153748877 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129715737850880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37398449 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89979315 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129715755853824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6266554224640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6177516887961 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6196192141824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37398451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157347133880320 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153748877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚735 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128176377200537 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129715737850880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63769562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121998877351936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153722668 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135912181594112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151080847276544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157347425441280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135912181594112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24228013 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6196425740288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61626462 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135911929992704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89953108 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151080871216640 _step‚276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0e¡{ë&ŸåM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135911929992704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89953108 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151080871216640 _step‚276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883986344791481e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6266286603776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6177745858017 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63769560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6196192141824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153722668 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128176623209953 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129715755853824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚737 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121998860312576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63769562 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61627199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63769560 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153748877 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129715737850880 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37398449 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89979315 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129715755853824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6266554224640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6177516887961 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6196192141824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37398451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157347133880320 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153748877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚735 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128176377200537 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129715737850880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63769562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121998877351936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153722668 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135912181594112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151080847276544 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157347425441280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135912181594112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24228013 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6196425740288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61626462 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3399”ÝpÙ2 - _wandbruntime‚3698  _runtime‚3698·`ê¯U¡Ï 1loss/SRqqt1/detection_loss‚3.0307928634726484 /loss/FRt3/assignment_loss‚5.759022328911758 .loss/FRt4/assignment_loss‚2.62268928634493 2loss/SRqqt2/assignment_loss‚2.3322218044384107 1loss/SRqqt3/detection_loss‚2.5497121775851532 /loss/FBt2/detection_loss‚2.7814586758613586 /loss/FRt2/detection_loss‚2.1089276383000035 /loss/FBt1/detection_loss‚1.7714318011341423 0loss/FBt3/assignment_loss‚0.7677927327652773 -loss/FRt3/detection_loss‚2.43809154466158 /loss/FBt3/detection_loss‚2.6827514370282493 %loss/total_loss‚55.65963921035232 $ _timestamp‚1.788398634480837e+09 _step‚277 /loss/FRt2/assignment_loss‚4.178239355164189 2loss/SRqqt1/assignment_loss‚2.6399525326231252 2loss/SRqqt4/assignment_loss‚2.0579696962875977 0loss/FBt1/assignment_loss‚0.4793011864338292 1loss/SRqqt4/detection_loss‚1.4470224490921413 1loss/FBt4/assignment_loss‚0.27366050981823353 /loss/FRt4/detection_loss‚2.3126485669299175 1loss/SRqqt2/detection_loss‚1.4531301495593947 epoch‚2 _runtime‚3698.040579111 -loss/FRt1/assignment_loss‚5.4274654325686 2loss/SRqqt3/assignment_loss‚2.6534569316050587 .loss/FRt1/detection_loss‚2.552837450253336 /loss/FBt4/detection_loss‚0.9150342927092597 0loss/FBt2/assignment_loss‚0.4240283668041229 trainer/global_step‚3399ï¸{›Z¢Ô 0loss/FBt3/assignment_loss‚0.7677927327652773 -loss/FRt3/detection_loss‚2.43809154466158 /loss/FBt3/detection_loss‚2.6827514370282493 %loss/total_loss‚55.65963921035232 $ _timestamp‚1.788398634480837e+09 _step‚277 /loss/FRt2/assignment_loss‚4.178239355164189 2loss/SRqqt1/assignment_loss‚2.6399525326231252 2loss/SRqqt4/assignment_loss‚2.0579696962875977 0loss/FBt1/assignment_loss‚0.4793011864338292 1loss/SRqqt4/detection_loss‚1.4470224490921413 1loss/FBt4/assignment_loss‚0.27366050981823353 /loss/FRt4/detection_loss‚2.3126485669299175 1loss/SRqqt2/detection_loss‚1.4531301495593947 epoch‚2 _runtime‚3698.040579111 -loss/FRt1/assignment_loss‚5.4274654325686 2loss/SRqqt3/assignment_loss‚2.6534569316050587 .loss/FRt1/detection_loss‚2.552837450253336 /loss/FBt4/detection_loss‚0.9150342927092597 0loss/FBt2/assignment_loss‚0.4240283668041229 trainer/global_step‚3399 1loss/SRqqt1/detection_loss‚3.0307928634726484 /loss/FRt3/assignment_loss‚5.759022328911758 .loss/FRt4/assignment_loss‚2.62268928634493 2loss/SRqqt2/assignment_loss‚2.3322218044384107 1loss/SRqqt3/detection_loss‚2.5497121775851532 /loss/FBt2/detection_loss‚2.7814586758613586 /loss/FRt2/detection_loss‚2.1089276383000035 /loss/FBt1/detection_loss‚1.7714318011341423•ì< }£‚(:… ±˜ãÔ«§àÓ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.565' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.113 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03^ÙDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91234135 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137820816473088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91260316 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37920446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6284854828032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24563624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137820564884992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131535710056960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6355905565184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123710285327360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159562604699648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚3743.926952137 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64660333 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64660333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131535728059904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155894466 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91234135 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6356166907904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6285088413184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155920649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 _step‚279 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123710268288000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91260316 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 % _timestamp‚1.7883986803669655e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504âãÕá'®ÛO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123710268288000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91260316 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 % _timestamp‚1.7883986803669655e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131535710056960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24564339 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6285088413184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64660331 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153206437791744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6266142387577 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 285282816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129976181705729 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137820564884992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153206413851648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64660331 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159562319416832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62484787 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37920448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91234135 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137820816473088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91260316 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37920446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6284854828032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24563624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137820564884992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131535710056960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6355905565184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123710285327360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159562604699648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚3743.926952137 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64660333 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64660333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131535728059904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155894466 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91234135 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6356166907904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6285088413184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155920649 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 _step‚279—SŠ³2¯- _wandbruntime‚3744  _runtime‚3744p+ãæ&°àM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62502184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281075200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153249792761344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37931101 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155964929 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131572852346880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6286663976448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _runtime‚3744.818220134 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91286417 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91260210 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 _step‚280 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137859516323328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6286663976448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6286897574912 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64678512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123745212020736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159607809959424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155964929 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚727 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62501457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24571083 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6267717151169 trainer/global_step‚3449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64678510 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131572870349824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130013158141961 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123745194981376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6286897574912 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137859516323328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6286663976448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6286897574912 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64678512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚725 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123745212020736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159607809959424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155964929 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚727 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62501457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24571083 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6267717151169 trainer/global_step‚3449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64678510 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131572870349824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130013158141961 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123745194981376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6286897574912 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 281075200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153249792761344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37931101 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155964929 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131572852346880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6286663976448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _runtime‚3744.818220134 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91286417 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91260210 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 _step‚280 EU[³Õ 1loss/FBt2/assignment_loss‚0.12755648652091622 /loss/FRt1/detection_loss‚2.5367130246655694 1loss/SRqqt4/detection_loss‚1.3361890955349884 epoch‚2 1loss/SRqqt3/detection_loss‚2.9221930344899496 /loss/FBt1/detection_loss‚1.6011789513589896 /loss/FRt3/detection_loss‚2.8958457186818123 0loss/SRqqt2/detection_loss‚1.450936900911393 /loss/FRt3/assignment_loss‚5.574086203294642 2loss/SRqqt1/assignment_loss‚3.0239488790112157 2loss/SRqqt2/assignment_loss‚2.2379378858324768 1loss/FBt4/assignment_loss‚0.24601515931762793 0loss/FRt4/assignment_loss‚2.4782119075457256 .loss/FRt2/detection_loss‚2.033723114545529 1loss/SRqqt1/detection_loss‚3.0287948269997873 .loss/FBt2/detection_loss‚5.364161213239034 trainer/global_step‚3449 /loss/FRt4/detection_loss‚1.8853199432293575 /loss/FBt4/detection_loss‚0.9811981054132476 0loss/FRt2/assignment_loss‚3.8630686209752008 1loss/SRqqt3/assignment_loss‚2.396008026599884 2loss/SRqqt4/assignment_loss‚1.9620643905489832 1loss/FBt1/assignment_loss‚0.47351251912004955 0loss/FBt3/assignment_loss‚0.5775147990709228 .loss/FBt3/detection_loss‚1.884203052520752 %loss/total_loss‚56.66547424613418 _step‚281 % _timestamp‚1.7883986812595553e+09 _runtime‚3744.819253241 /loss/FRt1/assignment_loss‚5.785092386706122 ©ò`´Ú .loss/FRt2/detection_loss‚2.033723114545529 1loss/SRqqt1/detection_loss‚3.0287948269997873 .loss/FBt2/detection_loss‚5.364161213239034 trainer/global_step‚3449 /loss/FRt4/detection_loss‚1.8853199432293575 /loss/FBt4/detection_loss‚0.9811981054132476 0loss/FRt2/assignment_loss‚3.8630686209752008 1loss/SRqqt3/assignment_loss‚2.396008026599884 2loss/SRqqt4/assignment_loss‚1.9620643905489832 1loss/FBt1/assignment_loss‚0.47351251912004955 0loss/FBt3/assignment_loss‚0.5775147990709228 .loss/FBt3/detection_loss‚1.884203052520752 %loss/total_loss‚56.66547424613418 _step‚281 % _timestamp‚1.7883986812595553e+09 _runtime‚3744.819253241 /loss/FRt1/assignment_loss‚5.785092386706122 1loss/FBt2/assignment_loss‚0.12755648652091622 /loss/FRt1/detection_loss‚2.5367130246655694 1loss/SRqqt4/detection_loss‚1.3361890955349884 epoch‚2 1loss/SRqqt3/detection_loss‚2.9221930344899496 /loss/FBt1/detection_loss‚1.6011789513589896 /loss/FRt3/detection_loss‚2.8958457186818123 0loss/SRqqt2/detection_loss‚1.450936900911393 /loss/FRt3/assignment_loss‚5.574086203294642 2loss/SRqqt1/assignment_loss‚3.0239488790112157 2loss/SRqqt2/assignment_loss‚2.2379378858324768 1loss/FBt4/assignment_loss‚0.24601515931762793 0loss/FRt4/assignment_loss‚2.4782119075457256™9EDºµ‚(:… Þ˜ãÔ·«ÍÔ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.326' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4652 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛéÍôD¶‚(:¹ Þ˜ãÔë–·Ö disk.sda3.in‚ 0.07421875' disk.nvme0c0n1.out‚49924.8193359375 disk.rbd0.out‚0 disk.sda.out‚ 3070.23046875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚2766.375 disk.dm-1.out‚ 3070.23046875 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚70.73898986349433% disk.nvme0n1.out‚49924.8193359375" disk.dm-0.out‚49924.8193359375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125 disk.dm-1.in‚ 0.07421875 disk.drbd1777.out‚0 network.recv‚ 1.33198e+06 disk.nvme0n1.in‚ 670.1484375 disk.dm-0.in‚ 670.1484375 disk.rbd3.in‚0$ disk./.usageGB‚1054.062068939209! disk.rbd2.out‚119.16162109375$ memory_percent‚98.37884902954102& proc.memory.availableMB‚ 265.609375 network.sent‚ 7.861453e+06 disk.rbd0.in‚0 disk.sda.in‚ 0.07421875* proc.memory.percent‚16.884613037109375 proc.cpu.threads‚17 disk.sda3.out‚ 3070.23046875! disk.nvme0c0n1.in‚ 670.1484375 cpu‚25.22097005309785L_‘·‚(:† í˜ãÔ‡áÓÒ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.488' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F6¸‚(:« í˜ãÔ¶œÖ disk.drbd1777.in‚0 disk.drbd1777.out‚0" disk.nvme0n1.out‚ 49925.1953125( proc.memory.availableMB‚ 265.51953125 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 670.1484375 disk.dm-0.in‚ 670.1484375 disk.dm-0.out‚ 49925.1953125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0$ memory_percent‚98.37939739227295$ disk./.usageGB‚1054.062183380127 disk.sda1.in‚0! disk.rbd2.out‚119.17724609375 disk.rbd3.out‚0 disk.sda.in‚ 0.07421875 disk.sda3.out‚ 3070.84375 disk.rbd1.in‚0 disk.dm-1.out‚ 3070.84375 disk.nvme0n1.in‚ 670.1484375 proc.memory.rssMB‚2766.375* proc.memory.percent‚16.884613037109375 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 49925.1953125 disk.dm-1.in‚ 0.07421875) disk./.usagePercent‚70.73899754372034 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.90283203125 cpu‚25.220296490682593 network.recv‚ 1.337997e+06 disk.sda.out‚ 3070.84375 disk.sda2.in‚0 network.sent‚ 7.969191e+06 disk.rbd0.in‚0ÖÛÇœ¹‚(:‘ ü˜ãÔ’­öÒ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.752' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.350399999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%}rûEº‚(:º ü˜ãÔœÚŸÖ disk.dm-0.in‚ 670.1484375 disk.rbd3.in‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 670.1484375( disk.nvme0c0n1.out‚49926.75341796875 disk.sda2.in‚0$ memory_percent‚98.38907718658447 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 network.recv‚ 1.361678e+06$ disk./.usageGB‚1054.062068939209 disk.sda3.out‚ 3071.6484375 disk.dm-1.in‚ 0.07421875 disk.sda1.out‚0 disk.rbd2.out‚ 119.326171875( proc.memory.availableMB‚ 263.93359375 cpu‚25.219625009275063 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 network.sent‚ 7.98513e+06 disk.sda2.out‚0 disk.dm-1.out‚ 3071.6484375 disk.sda.out‚ 3071.6484375 proc.memory.rssMB‚2766.375* proc.memory.percent‚16.884613037109375 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.90283203125# disk.dm-0.out‚49926.75341796875 disk.drbd1777.out‚0) disk./.usagePercent‚70.73898986349433 disk.rbd3.out‚0 disk.nvme0n1.in‚ 670.1484375& disk.nvme0n1.out‚49926.75341796875½æ9/2»- _wandbruntime‚3790  _runtime‚37909õ2ɼà _step‚282 ' lr-AdamW/pg1‚0.00019951629938367264 ' lr-AdamW/pg2‚0.00019951629938367264 trainer/global_step‚3499 % _timestamp‚1.7883987266700227e+09 _runtime‚3790.229855898j±¨/Î½È _step‚282 ' lr-AdamW/pg1‚0.00019951629938367264 ' lr-AdamW/pg2‚0.00019951629938367264 trainer/global_step‚3499 % _timestamp‚1.7883987266700227e+09 _runtime‚3790.229855898š!’ð2¾- _wandbruntime‚3790  _runtime‚3790$tQÜ'¿ÖO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65569281 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139768151215616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139768151215616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65569283 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161823033911808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158110518 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 % _timestamp‚1.7883987266707284e+09 _runtime‚3790.230723532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155375359336448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65569283 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63363359 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38453098 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6375326662656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133392842555904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133392824552960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6356113680937 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6375326662656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92567418 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125456602956800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92541237 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚742 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 287379968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960¼x«ßÀÛO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139768402803712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139768151215616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139768151215616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65569283 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161823033911808 UGDevicåbùÍÞeStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158110518 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 % _timestamp‚1.7883987266707284e+09 _runtime‚3790.230723532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155375359336448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65569283 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63363359 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38453098 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6375326662656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133392842555904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133392824552960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6356113680937 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6375326662656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92567418 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125456602956800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92541237 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚742 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 287379968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65569281 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63380042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6377369409536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6377135811072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125491529650176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92567312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158154772 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 trainer/global_step‚3499 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133429966842880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6449473831424 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161868236017664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161867961233920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139807354255360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65587462 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92567312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92593519 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133429966842880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6357917414377 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _step‚284 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6377369409536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92593519 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131849693073969 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158180981 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 250843648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6377135811072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65587460 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38463751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125491546689536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158180981 % _timestamp‚1.7883987275480468e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚735 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24917026 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6449222987776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032–Ûë&ÃåM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 _runtime‚3791.109429207 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158154772 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139807102653952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24916293 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6358146384433 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65587462 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92567312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92593519 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133429966842880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6357917414377 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _step‚284 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6377369409536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92593519 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131849693073969 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158180981 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 250843648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6377135811072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65587460 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38463751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125491546689536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158180981 % _timestamp‚1.7883987275480468e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚735 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24917026 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6449222987776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133429984845824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155418762186240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139807354255360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63380777 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65587462 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133429984845824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63380042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6377369409536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6377135811072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125491529650176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92567312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158154772 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 trainer/global_step‚3499 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133429966842880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6449473831424 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161868236017664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161867961233920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139807354255360 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66478233 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6466032082432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6465798497280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164083119675904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6466032082432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚141715989134336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64241368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6446542913993 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141715737546240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133649497578953 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93874520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64240638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25255620 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25254892 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135249957051904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141715737546240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127202937625600 _step‚287 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135249957051904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0½}µÎá'ÒÛO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66478231 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66478233 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6466032082432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6465798497280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164083119675904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6466032082432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚141715989134336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64241368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6446542913993 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141715737546240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133649497578953 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93874520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64240638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25255620 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25254892 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135249957051904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141715737546240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127202937625600 _step‚287 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135249957051904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160326570 % _timestamp‚1.7883987736327043e+09ŸS¯£2Ó- _wandbruntime‚3838  _runtime‚3838ÁF,àå&ÔßM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93900621 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164128613050368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6467841244160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6540905379328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66496410 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66496412 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157587707671040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 $ _timestamp‚1.788398774536383e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66496412 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93900621 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64259016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93874414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160370824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164128321489408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160397033 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚141754940585984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135287099341824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160370824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38996401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25262615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133686228005977 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127237864318976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135287081338880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6448346647641 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64258302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚141754688984576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 _runtime‚3838.096971038 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135287099341824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135287081338880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160397033 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133685981996561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¨ì³aIÕäM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127237864318976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135287081338880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6448346647641 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64258302 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚141754688984576 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 _runtime‚3838.096971038 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚135287099341824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135287081338880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160397033 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133685981996561 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93900621 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164128613050368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6467841244160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6540905379328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66496410 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚288 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66496412 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157587707671040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 $ _timestamp‚1.788398774536383e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66496412 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93900621 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64259016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93874414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160370824 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164128321489408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160397033 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚141754940585984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚135287099341824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160370824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38996401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25262615 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133686228005977 Ï6Ë2Ö- _wandbruntime‚3838  _runtime‚3838o]ëh\×Ö 0loss/FRt4/assignment_loss‚2.7744762460390726 0loss/SRqqt3/detection_loss‚2.561389360576868 /loss/FBt1/detection_loss‚1.8477963262796402 2loss/SRqqt2/assignment_loss‚2.4030613474547864 1loss/FBt3/assignment_loss‚0.35466350047742407 .loss/FRt4/detection_loss‚2.213066112664011 %loss/total_loss‚55.99255071335598 2loss/SRqqt1/assignment_loss‚2.5404072028619273 1loss/FBt1/assignment_loss‚0.38065224158781347 0loss/FBt2/assignment_loss‚0.4462619584852031 /loss/FBt3/detection_loss‚2.9621932903925576 /loss/FRt1/detection_loss‚2.4968003247465407 /loss/FRt3/assignment_loss‚5.574971665276421 1loss/FBt4/assignment_loss‚0.27091227975397564 1loss/SRqqt1/detection_loss‚2.5657862248244108 epoch‚2 % _timestamp‚1.7883987745382223e+09 _step‚289 /loss/FRt1/assignment_loss‚5.599589484078543 2loss/SRqqt4/assignment_loss‚2.1488210440113926 /loss/FRt2/detection_loss‚2.3387446204821267 /loss/FBt2/detection_loss‚3.4820971318653653 /loss/FBt4/detection_loss‚1.1238927515381474 /loss/FRt2/assignment_loss‚4.134955665644477 /loss/FRt3/detection_loss‚2.3591740290323893 0loss/SRqqt2/detection_loss‚1.444149000570178 _runtime‚3838.097928661 2loss/SRqqt3/assignment_loss‚2.7902894215658307 1loss/SRqqt4/detection_loss‚1.1783994831468747 trainer/global_step‚3549 ýŸ„aØÛ 1loss/SRqqt1/detection_loss‚2.5657862248244108 epoch‚2 % _timestamp‚1.7883987745382223e+09 _step‚289 /loss/FRt1/assignment_loss‚5.599589484078543 2loss/SRqqt4/assignment_loss‚2.1488210440113926 /loss/FRt2/detection_loss‚2.3387446204821267 /loss/FBt2/detection_loss‚3.4820971318653653 /loss/FBt4/detection_loss‚1.1238927515381474 /loss/FRt2/assignment_loss‚4.134955665644477 /loss/FRt3/detection_loss‚2.3591740290323893 0loss/SRqqt2/detection_loss‚1.444149000570178 _runtime‚3838.097928661 2loss/SRqqt3/assignment_loss‚2.7902894215658307 1loss/SRqqt4/detection_loss‚1.1783994831468747 trainer/global_step‚3549 0loss/FRt4/assignment_loss‚2.7744762460390726 0loss/SRqqt3/detection_loss‚2.561389360576868 /loss/FBt1/detection_loss‚1.8477963262796402 2loss/SRqqt2/assignment_loss‚2.4030613474547864 1loss/FBt3/assignment_loss‚0.35466350047742407 .loss/FRt4/detection_loss‚2.213066112664011 %loss/total_loss‚55.99255071335598 2loss/SRqqt1/assignment_loss‚2.5404072028619273 1loss/FBt1/assignment_loss‚0.38065224158781347 0loss/FBt2/assignment_loss‚0.4462619584852031 /loss/FBt3/detection_loss‚2.9621932903925576 /loss/FRt1/detection_loss‚2.4968003247465407 /loss/FRt3/assignment_loss‚5.574971665276421 1loss/FBt4/assignment_loss‚0.27091227975397564¡Jº¤›Ù‚(: ¸™ãÔƒèŒÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.051' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.610200000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û§ÇµBÚ‚(:· ¸™ãÔª·¨Õ disk.rbd0.in‚0( proc.memory.availableMB‚ 265.31640625( disk./.usagePercent‚70.7361573961355 disk.sda3.out‚ 3074.07421875% disk.nvme0n1.out‚50064.0400390625 disk.dm-0.in‚ 670.1484375$ memory_percent‚98.38063716888428 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 670.1484375 disk.sda1.out‚0 network.recv‚ 1.380334e+06$ disk./.usageGB‚1054.019863128662 disk.rbd2.in‚0 disk.rbd1.out‚ 1.90283203125 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 3074.07421875 disk.rbd2.out‚ 121.205078125 disk.rbd3.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚50064.0400390625 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0" disk.dm-0.out‚50064.0400390625 network.sent‚ 8.1107e+06 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 670.1484375 disk.sda.in‚ 0.07421875 proc.memory.rssMB‚2766.375* proc.memory.percent‚16.884613037109375 cpu‚25.21694067895176 disk.dm-1.out‚ 3074.07421875 disk.drbd1777.out‚0 disk.sda3.in‚ 0.07421875tô1ÚÛ‚(:„ Ç™ãÔŠøëÒ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.596 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ/æ@Ü‚(:µ Ç™ãÔí‡üÔ% disk./.usageGB‚1052.8239059448242! disk.nvme0c0n1.in‚ 670.1484375 disk.rbd0.in‚0 cpu‚25.21630027677405) disk./.usagePercent‚70.65589570605322 disk.sda.out‚ 3074.6015625 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.90283203125 disk.dm-1.out‚ 3074.6015625 network.sent‚ 8.218157e+06 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚98.38013648986816 proc.memory.rssMB‚2766.375 disk.dm-0.in‚ 670.1484375 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3074.6015625' disk.nvme0c0n1.out‚50065.2255859375* proc.memory.percent‚16.884613037109375 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0' proc.memory.availableMB‚ 265.3984375 network.recv‚ 1.3859e+06 disk.nvme0n1.in‚ 670.1484375% disk.nvme0n1.out‚50065.2255859375" disk.dm-0.out‚50065.2255859375 disk.dm-1.in‚ 0.07421875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd2.out‚ 121.216796875 disk.sda2.in‚0}¹ŽÝ‚(:„ Ö™ãÔ¾†œÓ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚29.0 gpu.0.powerWatts‚80.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.101 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UMÂJÞ‚(:¿ Ö™ãÔÞï¤Ö network.sent‚ 8.224468e+06# disk./.usageGB‚1052.82320022583 disk.sda3.in‚ 0.07421875! disk.nvme0c0n1.in‚ 670.1484375( disk.nvme0c0n1.out‚50201.24072265625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 265.12890625 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.884613037109375 network.recv‚ 1.390653e+06) disk./.usagePercent‚70.65584834465938 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 3075.88671875 disk.sda.out‚ 3075.88671875 disk.drbd1777.out‚0! disk.rbd2.out‚121.36572265625 disk.rbd3.out‚0 disk.dm-1.out‚ 3075.88671875 disk.dm-0.in‚ 670.1484375 disk.sda.in‚ 0.07421875 disk.rbd0.out‚0& disk.nvme0n1.out‚50201.24072265625# disk.dm-0.out‚50201.24072265625$ memory_percent‚98.38178157806396 proc.memory.rssMB‚2766.375 proc.cpu.threads‚17 disk.dm-1.in‚ 0.07421875 disk.sda2.in‚0 disk.rbd1.out‚ 1.90283203125 disk.nvme0n1.in‚ 670.1484375 cpu‚25.215578109743547yã<ß‚(:„ å™ãÔÚ¬éÒ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.03' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.006 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÈKz?à‚(:´ å™ãÔýšœÕ proc.cpu.threads‚17 network.recv‚ 1.393764e+06 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125 disk.drbd1777.in‚0$ disk.nvme0n1.out‚50201.580078125 disk.sda.out‚ 3076.234375$ memory_percent‚98.38025569915771( proc.memory.availableMB‚ 265.37890625 proc.memory.rssMB‚2766.375* proc.memory.percent‚16.884613037109375! disk.nvme0c0n1.in‚ 670.1484375$ disk./.usageGB‚1052.823314666748 disk.rbd2.in‚0 disk.sda3.out‚ 3076.234375 network.sent‚ 8.229009e+06) disk./.usagePercent‚70.65585602488541! disk.rbd2.out‚121.38134765625& disk.nvme0c0n1.out‚50201.580078125 disk.dm-1.out‚ 3076.234375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.1484375 disk.dm-0.in‚ 670.1484375 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0! disk.dm-0.out‚50201.580078125 cpu‚25.215021452388452¼½+Ö2á- _wandbruntime‚3885  _runtime‚3885Ñv_ÇâÁ & lr-AdamW/pg2‚0.0001975030000939961 trainer/global_step‚3599 % _timestamp‚1.7883988215031958e+09 _runtime‚3885.063004637 _step‚290 & lr-AdamW/pg1‚0.0001975030000939961V_ZäÌãÆ & lr-AdamW/pg2‚0.0001975030000939961 trainer/global_step‚3599 % _timestamp‚1.7883988215031958e+09 _runtime‚3885.063004637 _step‚290 & lr-AdamW/pg1‚0.0001975030000939961¢ $€ 2ä- _wandbruntime‚3885  _runtime‚3885$/Ç€Ü'åÖO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143663323876864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3885.063976928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95181622 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95181622 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137107071547904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137107053544960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6630207980032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245568 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522163712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159713274246144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135485786501753 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143663323876864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137107071547904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137107053544960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67387183 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159713250306048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6630467225600 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 _step‚291 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67387181 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67387183 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65119727 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39518398 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128949289333760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95155441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25601331 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162568805 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3599 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 % _timestamp‚1.7883988215039263e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65120425 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39518396 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95155441 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143663575464960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166343741471744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6536514207353 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6536743177201 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128949272294400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143663323876864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137107071547904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26183 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137107053544960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31807 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67387183 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159713250306048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6630467225600 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 _step‚291 L?DeviceStatsMonitor.on_train_batch_start/active.larg›àPe_pool.freed‚67387181 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67387183 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65119727 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39518398 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26181 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128949289333760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95155441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25601331 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162568805 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3599 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 % _timestamp‚1.7883988215039263e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522163712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65120425 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39518396 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95155441 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143663575464960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166343741471744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6536514207353 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚698 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6536743177201 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26181 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128949272294400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26183 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128984198987776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39529049 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6632034562048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135522762937985 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143702526916608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137144213837824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6558313078784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95207723 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95181516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137144195834880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65137852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25608116 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128984216027136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159756653155840 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162613085 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137144213837824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚292 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95207723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7883988224169595e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6538546910849 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143702526916608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162613085 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65137165 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6558079480320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6538317940793 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6558313078784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162586876 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95181516 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166388663777792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269718016 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6558079480320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166388957435904 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 _runtime‚3885.977629185â¥Ý”ë&éåM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95207723 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26209 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95181516 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137144195834880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65137852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25608116 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128984216027136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522163712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159756653155840 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162613085 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137144213837824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504845 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚292 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95207723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7883988224169595e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6538546910849 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143702526916608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162613085 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65137165 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6558079480320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6538317940793 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6558313078784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162586876 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95181516 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166388663777792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 269718016 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26207 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6558079480320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31807 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166388957435904 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 _runtime‚3885.977629185 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162586876 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137144195834880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67405362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128984198987776 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39529049 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6632034562048 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135522762937985 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354437 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143702526916608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26207 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137144213837824 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522163712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6558313078784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32¤Ö§U=2ê- _wandbruntime‚3885  _runtime‚3885æ÷xWëÑ .loss/FBt2/detection_loss‚3.455058753490448 1loss/FBt2/assignment_loss‚0.15758033888414502 2loss/SRqqt1/assignment_loss‚2.9136480406710974 /loss/FBt4/detection_loss‚1.1521394979504367 _runtime‚3885.978722685 /loss/FRt4/assignment_loss‚2.520373837514357 2loss/SRqqt4/assignment_loss‚1.9902976507381085 0loss/FRt1/assignment_loss‚6.0551737104143415 /loss/FRt3/assignment_loss‚5.342021131515503 2loss/SRqqt2/assignment_loss‚2.2344211408461647 0loss/FBt1/assignment_loss‚0.6645529901936061 0loss/SRqqt1/detection_loss‚2.927238668266096 2loss/SRqqt3/assignment_loss‚2.7596953367364816 /loss/FRt1/detection_loss‚2.7594432626451764 /loss/FRt2/detection_loss‚2.3249342693732333 /loss/SRqqt3/detection_loss‚2.63156010364664 epoch‚2 0loss/FBt4/assignment_loss‚0.2773890149236765 /loss/FRt3/detection_loss‚2.6194896530359983 1loss/SRqqt2/detection_loss‚1.3567474851834362 trainer/global_step‚3599 % _timestamp‚1.7883988224189885e+09 _step‚293 /loss/FBt1/detection_loss‚1.6191404867917298 /loss/FBt3/detection_loss‚2.6504776857116004 %loss/total_loss‚56.88418839945903 /loss/FRt2/assignment_loss‚4.511981617945891 /loss/FBt3/assignment_loss‚0.508835093552162 /loss/FRt4/detection_loss‚2.1671340702609583 0loss/SRqqt4/detection_loss‚1.284854559167739×Ê€\ìÖ 2loss/SRqqt4/assignment_loss‚1.9902976507381085 0loss/FRt1/assignment_loss‚6.0551737104143415 /loss/FRt3/assignment_loss‚5.342021131515503 2loss/SRqqt2/assignment_loss‚2.2344211408461647 0loss/FBt1/assignment_loss‚0.6645529901936061 0loss/SRqqt1/detection_loss‚2.927238668266096 2loss/SRqqt3/assignment_loss‚2.7596953367364816 /loss/FRt1/detection_loss‚2.7594432626451764 /loss/FRt2/detection_loss‚2.3249342693732333 /loss/SRqqt3/detection_loss‚2.63156010364664 epoch‚2 0loss/FBt4/assignment_loss‚0.2773890149236765 /loss/FRt3/detection_loss‚2.6194896530359983 1loss/SRqqt2/detection_loss‚1.3567474851834362 trainer/global_step‚3599 % _timestamp‚1.7883988224189885e+09 _step‚293 /loss/FBt1/detection_loss‚1.6191404867917298 /loss/FBt3/detection_loss‚2.6504776857116004 %loss/total_loss‚56.88418839945903 /loss/FRt2/assignment_loss‚4.511981617945891 /loss/FBt3/assignment_loss‚0.508835093552162 /loss/FRt4/detection_loss‚2.1671340702609583 0loss/SRqqt4/detection_loss‚1.284854559167739 .loss/FBt2/detection_loss‚3.455058753490448 1loss/FBt2/assignment_loss‚0.15758033888414502 2loss/SRqqt1/assignment_loss‚2.9136480406710974 /loss/FBt4/detection_loss‚1.1521394979504367 _runtime‚3885.978722685 /loss/FRt4/assignment_loss‚2.520373837514357¥ç ðí‚(:… ô™ãÔ§¤£Ó gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.7102 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý|òHî‚(:½ ô™ãÔîéÕ proc.memory.rssMB‚2766.375 network.sent‚ 8.235251e+06 network.recv‚ 1.398133e+06& disk.nvme0n1.out‚50202.47119140625* proc.memory.percent‚16.884613037109375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 265.1796875 disk.drbd1777.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 670.1484375 disk.dm-0.in‚ 670.1484375 disk.dm-1.in‚ 0.07421875 cpu‚25.21438296190463( disk./.usagePercent‚70.6558521847724$ disk./.usageGB‚1052.823257446289 disk.rbd2.in‚0! disk.rbd2.out‚123.08056640625# disk.dm-0.out‚50202.47119140625 disk.nvme0n1.in‚ 670.1484375 disk.sda.out‚ 3076.79296875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 3076.79296875 disk.rbd1.in‚0( disk.nvme0c0n1.out‚50202.47119140625 disk.dm-1.out‚ 3076.79296875$ memory_percent‚98.38147163391113 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.90283203125 disk.sda.in‚ 0.07421875ôËYЛï‚(: ƒšãÔÉÓ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚121.582' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.316399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù0‚ÛGð‚(:¼ ƒšãÔŠüÝÕ disk.rbd1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.90283203125! disk.nvme0c0n1.in‚ 670.1484375 disk.rbd0.in‚0& proc.memory.availableMB‚ 186.921875 proc.cpu.threads‚17 disk.dm-0.in‚ 670.1484375 cpu‚25.21731041566467 network.sent‚ 8.342717e+06 network.recv‚ 1.403602e+06 disk.sda2.in‚0( disk.nvme0c0n1.out‚50202.84619140625 disk.dm-1.out‚ 3077.23046875 disk.rbd0.out‚0 disk.sda.out‚ 3077.23046875* proc.memory.percent‚17.363739013671875 disk.nvme0n1.in‚ 670.1484375& disk.nvme0n1.out‚50202.84619140625 disk.sda2.out‚0# memory_percent‚98.8591194152832) disk./.usagePercent‚70.65640234496348$ disk./.usageGB‚1052.831455230713 disk.sda3.in‚ 0.07421875 proc.memory.rssMB‚2844.875 disk.dm-1.in‚ 0.07421875 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 3077.23046875 disk.sda.in‚ 0.07421875# disk.dm-0.out‚50202.84619140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚123.09619140625 disk.rbd3.in‚0Þàíõšñ‚(: ’šãÔù˜‰Ó gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.051' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.610200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06ªg.<ò‚(:± ’šãÔ±½ìÕ disk.sda3.out‚ 3077.84375 disk.dm-1.in‚ 0.07421875" proc.memory.rssMB‚ 2846.2734375) disk./.usagePercent‚70.65639517675251 disk.dm-0.in‚ 670.1484375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚ 1.408022e+06! disk.rbd2.out‚123.15869140625 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875! disk.nvme0c0n1.in‚ 670.1484375 disk.dm-1.out‚ 3077.84375( proc.memory.availableMB‚ 185.55859375 disk.drbd1777.out‚0 disk.sda2.in‚0% disk.nvme0c0n1.out‚50212.29296875$ memory_percent‚98.86744022369385) proc.memory.percent‚17.37227439880371 cpu‚25.23499862612247 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125 disk.rbd0.out‚0% disk./.usageGB‚1052.8313484191895 disk.dm-0.out‚50212.29296875 disk.sda2.out‚0# disk.nvme0n1.out‚50212.29296875 disk.sda.out‚ 3077.84375 network.sent‚ 8.348953e+06 disk.nvme0n1.in‚ 670.1484375 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.rbd3.out‚0oB7ÔŽó‚(:ƒ ¡šãÔþé÷Ò gpu.0.gpu‚24.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.34' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jŸj|ô‚(:¼ ¡šãÔ’¶™× disk.sda1.out‚0# memory_percent‚98.8673210144043 cpu‚25.25253683696311 network.re?pËcv‚ 1.412223e+06$ disk./.usageGB‚1052.831371307373 disk.sda2.out‚0" proc.memory.rssMB‚ 2846.2734375) proc.memory.percent‚17.37227439880371 network.sent‚ 8.354599e+06 disk.drbd1777.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚50212.68310546875& proc.memory.availableMB‚ 185.578125 disk.rbd2.in‚0! disk.rbd2.out‚123.17431640625 disk.rbd3.out‚0 disk.sda3.out‚ 3078.3359375 disk.dm-0.in‚ 670.1484375# disk.dm-0.out‚50212.68310546875 disk.sda1.in‚0) disk./.usagePercent‚70.65639671279772 disk.rbd1.out‚ 1.90283203125! disk.nvme0c0n1.in‚ 670.1484375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 670.1484375 disk.sda.in‚ 0.07421875 disk.sda.out‚ 3078.3359375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚50212.68310546875 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 3078.3359375 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0±ršõ‚(: °šãÔð‘ÕÓ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.237' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.647399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HI‚]Eö‚(:º °šãÔ輯Õ disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0! disk.nvme0c0n1.in‚ 670.1484375% disk./.usageGB‚1052.8316383361816 disk.rbd0.in‚0% disk.nvme0n1.out‚50213.2041015625 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 cpu‚25.267476355996052 network.recv‚ 1.41706e+06 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda3.in‚ 0.07421875' disk.nvme0c0n1.out‚50213.2041015625 network.sent‚ 8.360996e+06 disk.rbd2.in‚0 disk.sda3.out‚ 3078.9453125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.1484375 disk.dm-0.in‚ 670.1484375 disk.sda.out‚ 3078.9453125 disk.rbd1.out‚ 1.90283203125" disk.dm-0.out‚50213.2041015625 disk.dm-1.out‚ 3078.9453125# memory_percent‚98.8675594329834' proc.memory.availableMB‚ 185.5390625) proc.memory.percent‚17.37227439880371 proc.cpu.threads‚17) disk./.usagePercent‚70.65641463332511 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875" proc.memory.rssMB‚ 2846.2734375 disk.rbd2.out‚123.32324218750«`’2÷- _wandbruntime‚3964  _runtime‚3964 QŸ8C*ø½T QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31807 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193750016 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚146083583957504 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6651354437 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚68646875 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 319570432 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1382 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚131170634075136 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7035944960 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚146083777707520 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚139459704532480 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚169001590864896 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚162302968487936 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171811924 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6623897427968 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 164968514 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32032 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3516 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚169001432293376 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6874205184 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1354 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134631424 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚307 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚96301055 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚253 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6624073175040 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6624073175040 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 517504845 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚253 % _timestamp‚1.7883989008176532e+09 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚139459686529536 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚139459686529536 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193750016 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6698487745536 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚96301055 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175747072 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 164947928 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚560 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7035944960 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 530579456 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚307 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3516 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚139459704532480 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚146083583957504 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚399 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 164968514 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚137774544496889 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚68646875 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚66366986 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚3626 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚131170651114496 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚253 % _timestamp‚1.7883989008176532e+09 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚139459686529536 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚139459686529536 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193750016 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6698487745536 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚96301055 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175747072 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 164947928 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚560 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7035944960 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 530579456 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚307 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3516 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚139459704532480 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚146083583957504 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚399 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 164968514 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚137774544496889 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚68646875 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚66366986 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚3626 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚131170651114496 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32032 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚40275480 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚169029840009216 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚96309411 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚146110911221248 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6624242189312 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26107395 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1354 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚253 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚68662195 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚68662193 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6408446976 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚96309411 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚560 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚307 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚146110717471232 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 517504845 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6624242189312 _runtime‚3965.438824769 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175747072 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚253 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚399 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193750016 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 522163712 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6698832506880 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚131196009878528 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171811924 % _timestamp‚1.7883989018785334e+09 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 319570432 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6189583360 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32032 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚40275480 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚169029840009216 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚96309411 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚146110911221248 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 6624242189312 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26107395 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1354 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚253 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚68662195 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚68662193 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6408446976 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚96309411 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚560 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚307 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚146110717471232 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 517504845 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6624242189312 _runtime‚3965.438824769 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175747072 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚253§ÄL[bý‚(:‚ ¿šãÔùí…Ó gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.34' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cÿ7ŽBþ‚(:· ¿šãÔóÒÒÕ$ disk.nvme0n1.out‚50213.919921875! disk.dm-0.out‚50213.919921875 disk.sda.in‚ 0.07421875& disk.nvme0c0n1.out‚50213.919921875 disk.dm-0.in‚ 670.1484375" proc.memory.rssMB‚ 2846.2734375 disk.dm-1.out‚ 3079.359375 disk.rbd2.in‚0 disk.sda2.in‚0 network.sent‚ 8.366184e+06 disk.sda1.out‚0 disk.drbd1777.in‚0! disk.nvme0c0n1.in‚ 670.1484375 proc.cpu.threads‚17 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.sda.out‚ 3079.359375 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.1484375$ memory_percent‚98.87044429779053 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.37227439880371 disk.rbd0.in‚0 network.recv‚ 1.420481e+06 disk.sda3.out‚ 3079.359375( proc.memory.availableMB‚ 185.06640625 cpu‚25.284900277671564 disk.rbd2.out‚124.9130859375) disk./.usagePercent‚70.65639722481279% disk./.usageGB‚1052.8313789367676 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125á¦}2ÿ- _wandbruntime‚3979  _runtime‚3979#¸¿²eH€Ŧ particle/accuracy_1_of_4‚1 particle/accuracy_6_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1081412062048912 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2060444951057434 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2663973271846771 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9934802651405334 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08046421408653259 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 +jet/accuracy_2_of_4‚0.22768086194992065 +particle/specificity‚0.9247790575027466 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0793650820851326 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870588779449463 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.10252313315868378 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08127571642398834 particle/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872127771377563 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.18000000715255737 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.25 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.12884992361068726 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.23418231308460236 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0871647447347641 *jet/accuracy_1_of_3‚0.6736465692520142 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2321428507566452 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_4‚0.023484988138079643 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2430555522441864 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.26919642090797424 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.20404775440692902 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992769360542297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3336309492588043 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9861047863960266 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_3‚0.010496827773749828 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.125 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07092198729515076 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9898070096969604 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9808295369148254 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.23769865930080414 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0035714285913854837 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3573633134365082 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.019498735666275024 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11149424314498901 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07960743457078934 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12936587631702423 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.1533978134393692 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_5‚0.8474909663200378 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.17821350693702698 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 *loss/val_total_loss‚55.447162727419645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11929824203252792 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12511372566223145 ,jet/accuracy_3_of_6‚0.057692307978868484 jet/accuracy_6_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0037735849618911743 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.20727385580539703 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6156075596809387 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13522695004940033 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.18285289406776428 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3682001829147339 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988858103752136 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.04444444552063942 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.992368757724762 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.13955938816070557 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918791055679321 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 _step‚296 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0695091038942337 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04424859955906868 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1558641940355301 *jet/accuracy_2_of_6‚0.3076923191547394 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31355032324790955 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26121288537979126 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 trainer/global_step‚3626 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0015873016091063619 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1229797974228859 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.28814104199409485 particle/accuracy_3_of_3‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835831522941589 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.015115349553525448 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.013875321485102177 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05795055627822876 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.27005094289779663 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05744967237114906 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11531531065702438 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.01751055382192135 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2267267107963562 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0019229674944654107 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20484542846679688 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 jet/accuracy_2_of_7‚0 particle/accuracy_3_of_7‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007296673022210598 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24351854622364044 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006308645475655794 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.2253747582435608 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2301076352596283 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0023832430597394705 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9931973218917847 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3447522521018982 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894580245018005 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 $ _timestamp‚1.788398915927153e+09 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0690537616610527 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6449928283691406 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.065738245844841 particle/accuracy_4_of_5‚1 7validation_average_jet_accuracy‚0.31738078594207764 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <"Purity/2FRt0SRqqt1FBt/event_purity‚0.0071428571827709675 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9956803321838379 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4394471049308777 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.12413479387760162 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892371892929077 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09438589215278625 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 _runtime‚3979.489025036 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988858103752136 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9841616153717041 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_6‚0.8717949390411377 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.28897175192832947 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.19651801884174347 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06060606241226196 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.16788357496261597 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3324061334133148 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3149305284023285 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2271825522184372 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38485217094421387 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11611636728048325 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.27423855662345886 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9714862108230591 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26619499921798706 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.23144295811653137 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.14999999105930328 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2844891846179962 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt4SRqqt1dX?TçJFBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.33255016803741455 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2569444477558136 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11793448030948639 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3526649475097656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04424859955906868 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13599681854248047 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2638888657093048 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.18603606522083282 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9959464073181152 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278960824012756 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1406966745853424 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08396723866462708 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5303030610084534 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.32029011845588684 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.13333334028720856 epoch‚2 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4469750225543976 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16095677018165588 +jet/accuracy_2_of_3‚0.15256735682487488 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22145120799541473 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2286871373653412 particle/accuracy_5_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3990527391433716 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0760628879070282 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.03257168456912041 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.007834102027118206 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39444446563720703 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.17305052280426025 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.10870417952537537 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.28947368264198303 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.17396515607833862 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.160458043217659 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0048055038787424564 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 *"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9861317873001099 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 particle/accuracy_2_of_6‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08656910806894302 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.15533094108104706 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.166666641831398 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0009920635493472219 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11287950724363327 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1979166567325592 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.26620373129844666 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1875 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.24014681577682495 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03593577444553375 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2173043042421341 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.39924243092536926 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.24944105744361877 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22729884088039398 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2639898657798767 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3126618564128876 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.03742049261927605 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2691408693790436 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3276430666446686 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2848837077617645 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_1_of_7‚1 particle/accuracy_6_of_7‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3677821755409241 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012210012646391988 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.1982758641242981 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13070645928382874 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.25795719027519226 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09382112324237823 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.31030702590942383 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.2853540778160095 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_5‚0.005376344081014395 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23385770618915558 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940608739852905 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07001986354589462 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12223599851131439 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.20738928020000458 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8169642686843872 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.12962962687015533 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2396082580089569 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.28622013330459595 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05537689849734306 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.20437711477279663 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.15979258716106415 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.810458779335022 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3127133846282959 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.5771294832229614 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.30855268239974976 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3926492929458618 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2615789473056793 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12651190161705017 particle/accuracy_1_of_1‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.17870621383190155 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.37119174003601074 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0469403900206089 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12333767861127853 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2593750059604645 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.25928542017936707 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.18209876120090485 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2727011740207672 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08024179190397263 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.24944105744361877 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9914918541908264 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_2_of_5‚0.2632744312286377 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13766902685165405 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.19708117842674255 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07327834516763687 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17162972688674927 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7238667607307434 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.1899999976158142 jet/accuracy_1_of_7‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08081807941198349 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.24621213972568512 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962272047996521 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2148972600698471 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0763857439160347 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9909916520118713 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1875 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.30862700939178467 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13182666897773743 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.23392459750175476 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004560915287584066 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33368584513664246 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1Ó¡Ãý 5ʦ ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.007834102027118206 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39444446563720703 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.17305052280426025 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.10870417952537537 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.28947368264198303 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.17396515607833862 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.160458043217659 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0048055038787424564 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 *"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9861317873001099 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 particle/accuracy_2_of_6‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08656910806894302 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.15533094108104706 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.166666641831398 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0009920635493472219 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11287950724363327 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1979166567325592 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.26620373129844666 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1875 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.24014681577682495 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03593577444553375 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2173043042421341 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.39924243092536926 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.24944105744361877 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22729884088039398 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2639898657798767 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3126618564128876 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.03742049261927605 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2691408693790436 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3276430666446686 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2848837077617645 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_1_of_7‚1 particle/accuracy_6_of_7‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 & Purity/1FRt1SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3677821755409241 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012210012646391988 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.1982758641242981 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13070645928382874 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.25795719027519226 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09382112324237823 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.31030702590942383 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.2853540778160095 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_5‚0.005376344081014395 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23385770618915558 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940608739852905 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07001986354589462 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12223599851131439 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.20738928020000458 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8169642686843872 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.12962962687015533 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2396082580089569 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.28622013330459595 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05537689849734306 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.20437711477279663 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.15979258716106415 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.810458779335022 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3127133846282959 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.5771294832229614 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.30855268239974976 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3926492929458618 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2615789473056793 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12651190161705017 particle/accuracy_1_of_1‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.17870621383190155 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.37119174003601074 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0469403900206089 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12333767861127853 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2593750059604645 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.25928542017936707 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.18209876120090485 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2727011740207672 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08024179190397263 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.24944105744361877 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9914918541908264 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_2_of_5‚0.2632744312286377 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13766902685165405 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.19708117842674255 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07327834516763687 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17162972688674927 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7238667607307434 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.1899999976158142 jet/accuracy_1_of_7‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08081807941198349 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9722222089767456 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.24621213972568512 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962272047996521 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2148972600698471 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0763857439160347 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqÎÿÿF^qt*FBt/FBt_purity‚0.9909916520118713 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1875 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.30862700939178467 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13182666897773743 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.23392459750175476 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004560915287584066 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33368584513664246 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_4‚1 particle/accuracy_6_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1081412062048912 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2060444951057434 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2663973271846771 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9934802651405334 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08046421408653259 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 +jet/accuracy_2_of_4‚0.22768086194992065 +particle/specificity‚0.9247790575027466 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0793650820851326 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870588779449463 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.10252313315868378 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2142857164144516 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08127571642398834 particle/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872127771377563 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.18000000715255737 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.25 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.12884992361068726 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.23418231308460236 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0871647447347641 *jet/accuracy_1_of_3‚0.6736465692520142 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2321428507566452 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_4‚0.023484988138079643 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2430555522441864 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.26919642090797424 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.20404775440692902 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992769360542297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3336309492588043 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9861047863960266 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_3‚0.010496827773749828 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.125 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07092198729515076 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9898070096969604 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9808295369148254 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.23769865930080414 <"Purity/2FRt*SRqqt1FBt/event_purity‚0.0035714285913854837 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3573633134365082 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.019498735666275024 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11149424314498901 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07960743457078934 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12936587631702423 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.1533978134393692 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_5‚0.8474909663200378 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.17821350693702698 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 *loss/val_total_loss‚55.447162727419645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11929824203252792 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12511372566223145 ,jet/accuracy_3_of_6‚0.057692307978868484 jet/accuracy_6_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 <"Purity/0FRt2SRqqt1FBt/event_purity‚0.0037735849618911743 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.20727385580539703 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6156075596809387 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13522695004940033 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.18285289406776428 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3682001829147339 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9988858103752136 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.04444444552063942 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.992368757724762 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.13955938816070557 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918791055679321 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 _step‚296 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0695091038942337 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04424859955906868 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1558641940355301 *jet/accuracy_2_of_6‚0.3076923191547394 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31355032324790955 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26121288537979126 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 trainer/global_step‚3626 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0015873016091063619 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1229797974228859 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.28814104199409485 particle/accuracy_3_of_3‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835831522941589 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.015115349553525448 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.013875321485102177 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05795055627822876 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.27005094289779663 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05744967237114906 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11531531065702438 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.01751055382192135 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2267267107963562 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0019229674944654107 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20484542846679688 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 jet/accuracy_2_of_7‚0 particle/accuracy_3_of_7‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007296673022210598 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24351854622364044 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006308645475655794 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.2253747582435608 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2301076352596283 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0023832430597394705 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9931973218917847 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3447522521018982 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894580245018005 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 $ _timestamp‚1.788398915927153e+09 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0690537616610527 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6449928283691406 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.065738245844841 particle/accuracy_4_of_5‚1 7validation_average_jet_accuracy‚0.31738078594207764 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <"Purity/2FRt0SRqqt1FBt/event_purity‚0.0071428571827709675 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9956803321838379 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4394471049308777 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.12413479387760162 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892371892929077 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09438589215278625 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 _runtime‚3979.489025036 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9988858103752136 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9841616153717041 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_6‚0.8717949390411377 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.28897175192832947 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.19651801884174347 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06060606241226196 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.16788357496261597 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3324061334133148 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3149305284023285 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2271825522184372 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.38485217094421387 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11611636728048325 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.27423855662345886 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9714862108230591 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.26619499921798706 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.23144295811653137 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.14999999105930328 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2844891846179962 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.33255016803741455 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2569444477558136 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11793448030948639 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3526649475097656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04424859955906868 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13599681854248047 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2638888657093048 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.18603606522083282 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9959464073181152 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278960824012756 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.15000000596046448 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1406966745853424 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08396723866462708 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5303030610084534 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.32029011845588684 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.13333334028720856 epoch‚2 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4469750225543976 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16095677018165588 +jet/accuracy_2_of_3‚0.15256735682487488 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22145120799541473 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.2286871373653412 particle/accuracy_5_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3990527391433716 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0760628879070282 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.03257168456912041¨¼˜»@‚‚(:… ΚãÔðÉÖÔ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6874 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„–ôxLƒ‚(:Á ΚãÔŸûðÕ disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3080.12109375 cpu‚25.288937073760845 disk.sda1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚50385.2880859375* proc.memory.percent‚17.378592491149902) disk./.usagePercent‚70.67603940287738 disk.rbd3.out‚0 disk.nvme0n1.in‚ 670.1484375 disk.dm-0.in‚ 670.1484375 disk.sda.out‚ 3080.12109375% disk./.usageGB‚1053.1240615844727 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 3080.12109375# proc.memory.rssMB‚ 2847.30859375 network.recv‚ 1.4271e+06 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.07421875$ memory_percent‚97.87733554840088 proc.cpu.threads‚17" disk.dm-0.out‚50385.2880859375 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.rbd1.out‚ 1.90283203125! disk.nvme0c0n1.in‚ 670.1484375( proc.memory.availableMB‚ 347.77734375 disk.rbd0.in‚0 disk.rbd2.in‚0 network.sent‚ 8.474569e+06% disk.nvme0n1.out‚50385.2880859375! disk.rbd2.out‚124.98388671875 disk.rbd3.in‚0 disk.sda2.in‚0Aa 2„- _wandbruntime‚4003  _runtime‚4003¨úáÉ…Ã trainer/global_step‚3649 % _timestamp‚1.7883989401479945e+09 _runtime‚4003.707878529 _step‚297 ' lr-AdamW/pg1‚0.00019646936246299563 ' lr-AdamW/pg2‚0.000196469362462995638£VõÎ†È _step‚297 ' lr-AdamW/pg1‚0.00019646936246299563 ' lr-AdamW/pg2‚0.00019646936246299563 trainer/global_step‚3649 % _timestamp‚1.7883989401479945e+09 _runtime‚4003.707878529©5ŸG2‡- _wandbruntime‚4003  _runtime‚4003Œ®SÁ«ˆÖO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26335618 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138942877671562 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166252197 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96990906 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6668712706560 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96990906 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147320338562048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163650161924096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _runtime‚4003.709042392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚140651877444096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147320590150656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚138942631662350 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132294299616256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚140651859441152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97017088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69261293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293670912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147320590150656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6668712706560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚140651877444096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6743574157824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6668479120896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 _step‚298 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6648578055306 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6648349085454 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147320338562048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26334903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269730816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¢«—áá'‰ÛO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69261293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293670912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147320590150656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6668712706560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚140651877444096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6743574157824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6668479120896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 _step‚298 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6648578055306 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6648349085454 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147320338562048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26334903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269730816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26335618 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138942877671562 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166252197 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96990906 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6668712706560 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96990906 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147320338562048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163650161924096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _runtime‚4003.709042392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚140651877444096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147320590150656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚138942631662350 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132294299616256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚140651859441152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97017088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096ª‡ÑóF2Š- _wandbruntime‚4004  _runtime‚4004CðNÖæ&‹àM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69279472 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147359541602304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚140689001731072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40639787 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140689019734016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166296451 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97016981 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132329226309632 trainer/global_step‚3649 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97043189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26342108 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 % _timestamp‚1.7883989410300245e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6670288269312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163693540833792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6745667172352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4004.590356794 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚138979362089166 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6650152818894 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26341417 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6670521868288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69279470 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66981204 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138979608098586 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97043189 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6670521868288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166296451 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚140689019734016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166322661 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163693564773888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69279472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚140689001731072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40639789 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6745399552000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132329209270272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6650381788954 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208Mx}ß³"ŒåM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40639789 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6745399552000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132329209270272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6650381788954 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69279472 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147359541602304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚140689001731072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40639787 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140689019734016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166296451 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97016981 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132329226309632 trainer/global_step‚3649 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97043189 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26342108 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 % _timestamp‚1.7883989410300245e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6670288269312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163693540833792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6745667172352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4004.590356794 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚138979362089166 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6650152818894 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26341417 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6670521868288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69279470 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66981204 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138979608098586 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97043189 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6670521868288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 166296451 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.al†€Ô58located‚140689019734016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166322661 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163693564773888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69279472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚140689001731072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072«|™\2- _wandbruntime‚4004  _runtime‚4004Ü$ŠªOŽÉ 1loss/SRqqt4/detection_loss‚1.1481349836103618 0loss/FBt1/assignment_loss‚0.4759577935948038 epoch‚3 trainer/global_step‚3649 2loss/SRqqt3/assignment_loss‚2.7668203115463257 /loss/FBt3/assignment_loss‚0.515155865205952 0loss/FBt4/assignment_loss‚0.2736753226286776 ,loss/FBt2/detection_loss‚2.3425179819266 _step‚300 .loss/FRt2/assignment_loss‚3.69036559995852 2loss/SRqqt4/assignment_loss‚2.2255571966525167 /loss/FRt4/detection_loss‚2.0352655716088353 _runtime‚4004.591659347 0loss/FBt2/assignment_loss‚0.4597948242523368 % _timestamp‚1.7883989410318935e+09 /loss/FRt1/assignment_loss‚5.082760310942127 .loss/FRt3/assignment_loss‚5.29831025407121 -loss/FRt1/detection_loss‚3.04962823660143 2loss/SRqqt1/assignment_loss‚2.5360580108783863 0loss/SRqqt2/assignment_loss‚2.31445213280073 .loss/FRt3/detection_loss‚2.578025963258099 1loss/SRqqt1/detection_loss‚2.8332121592980846 1loss/SRqqt2/detection_loss‚1.5559738014892834 /loss/FBt1/detection_loss‚1.8294539573656108 .loss/FBt3/detection_loss‚3.135782814025879 %loss/total_loss‚54.94242006241511 /loss/FRt4/assignment_loss‚2.525567586324653 /loss/FBt4/detection_loss‚0.6878774263328169 /loss/FRt2/detection_loss‚2.6498514555002513 1loss/SRqqt3/detection_loss‚2.9322205025416155n“ÿTÎ 0loss/FBt2/assignment_loss‚0.4597948242523368 % _timestamp‚1.7883989410318935e+09 /loss/FRt1/assignment_loss‚5.082760310942127 .loss/FRt3/assignment_loss‚5.29831025407121 -loss/FRt1/detection_loss‚3.04962823660143 2loss/SRqqt1/assignment_loss‚2.5360580108783863 0loss/SRqqt2/assignment_loss‚2.31445213280073 .loss/FRt3/detection_loss‚2.578025963258099 1loss/SRqqt1/detection_loss‚2.8332121592980846 1loss/SRqqt2/detection_loss‚1.5559738014892834 /loss/FBt1/detection_loss‚1.8294539573656108 .loss/FBt3/detection_loss‚3.135782814025879 %loss/total_loss‚54.94242006241511 /loss/FRt4/assignment_loss‚2.525567586324653 /loss/FBt4/detection_loss‚0.6878774263328169 /loss/FRt2/detection_loss‚2.6498514555002513 1loss/SRqqt3/detection_loss‚2.9322205025416155 1loss/SRqqt4/detection_loss‚1.1481349836103618 0loss/FBt1/assignment_loss‚0.4759577935948038 epoch‚3 trainer/global_step‚3649 2loss/SRqqt3/assignment_loss‚2.7668203115463257 /loss/FBt3/assignment_loss‚0.515155865205952 0loss/FBt4/assignment_loss‚0.2736753226286776 ,loss/FBt2/detection_loss‚2.3425179819266 _step‚300 .loss/FRt2/assignment_loss‚3.69036559995852 2loss/SRqqt4/assignment_loss‚2.2255571966525167 /loss/FRt4/detection_loss‚2.0352655716088353 _runtime‚4004.591659347¬uzY‚(:… ÝšãÔ»ªÓ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚87.617' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.5234 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É°ìJ‘‚(:¿ ÝšãÔšÅéÖ disk.dm-1.in‚ 0.07421875 disk.sda1.in‚0$ memory_percent‚97.87631034851074 disk.sda.out‚ 3080.53515625$ disk.nvme0n1.out‚50492.435546875 disk.sda.in‚ 0.07421875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 3080.53515625' proc.memory.availableMB‚ 347.9453125) disk./.usagePercent‚70.67596311263218 disk.rbd2.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2846.32421875 disk.rbd1.out‚ 1.90283203125* proc.memory.percent‚17.372584342956543 proc.cpu.threads‚17 network.sent‚ 8.597989e+06% disk./.usageGB‚1053.1229248046875 disk.sda3.out‚ 3080.53515625& disk.nvme0c0n1.out‚50492.435546875 disk.rbd0.out‚0! disk.dm-0.out‚50492.435546875 disk.sda1.out‚0 network.recv‚ 1.432537e+06 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875! disk.nvme0c0n1.in‚ 670.1484375 disk.dm-0.in‚ 670.1484375 cpu‚25.287973981595357 disk.drbd1777.in‚0! disk.rbd2.out‚124.99951171875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 670.1484375 disk.drbd1777.out‚0]ÍUœ’‚(:‘ ìšãÔá—Ô gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.429' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.285800000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q0…rJ“‚(:¿ ìšãÔ™È¡Õ disk.rbd2.out‚ 125.1484375 disk.rbd3.out‚0( proc.memory.availableMB‚ 347.47265625# proc.memory.rssMB‚ 2846.32421875) disk./.usagePercent‚70.67595927251917! disk.nvme0c0n1.in‚ 670.1484375 disk.rbd3.in‚0 network.sent‚ 8.7066e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚50655.76123046875 disk.sda.out‚ 3081.08984375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3081.08984375 disk.dm-1.out‚ 3081.08984375 disk.rbd0.in‚0 disk.dm-0.in‚ 670.1484375# disk.dm-0.out‚50655.76123046875 network.recv‚ 1.438756e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 1.90283203125& disk.nvme0n1.out‚50655.76123046875 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd0.out‚0$ memory_percent‚97.87919521331787% disk./.usageGB‚1053.1228675842285 disk.rbd1.in‚0 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 670.1484375* proc.memory.percent‚17.372584342956543 cpu‚25.28705018497599q Ýœ”‚(:‘ ûšãÔ´¿âÒ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.437' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.687399999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°H<J•‚(:¿ ûšãÔ²®®Ö disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 670.1484375 disk.rbd2.in‚0 disk.rbd2.out‚ 125.1640625 disk.rbd3.in‚0$ memory_percent‚97.15516567230225 proc.cpu.threads‚17) disk./.usagePercent‚70.68441827346554 disk.rbd1.out‚ 1.90283203125( disk.nvme0c0n1.out‚50754.92138671875 disk.dm-0.in‚ 670.1484375 network.recv‚ 1.442592e+06% disk./.usageGB‚1053.2489128112793 disk.drbd1777.out‚0 disk.dm-1.out‚ 3081.4765625 disk.rbd0.out‚0 disk.sda.out‚ 3081.4765625* proc.memory.percent‚17.372584342956543 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 670.1484375 disk.sda1.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0& disk.nvme0n1.out‚50754.92138671875 disk.sda.in‚ 0.07421875( proc.memory.availableMB‚ 466.09765625 cpu‚25.286222156103474 network.sent‚ 8.711752e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.07421875# disk.dm-0.out‚50754.92138671875# proc.memory.rssMB‚ 2846.32421875 disk.sda3.out‚ 3081.4765625ƒ»Î:‘–‚(:† Š›ãÔÎŒðÒ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5564 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ûYêH—‚(:½ Š›ãÔµäßÖ disk.sda1.in‚0 disk.drbd1777.in‚0 network.sent‚ 8.718174e+06 disk.rbd0.in‚0$ memory_percent‚97.69129753112793 network.recv‚ 1.447057e+06 disk.rbd1.in‚0 disk.sda.out‚ 3082.00390625 disk.sda2.in‚0 disk.sda3.out‚ 3082.00390625' disk.nvme0c0n1.out‚50755.9150390625 disk.dm-1.out‚ 3082.00390625) disk./.usagePercent‚70.68442134555593 disk.rbd2.out‚ 125.2890625 disk.sda2.out‚0# proc.memory.rssMB‚ 2846.33984375 disk.rbd1.out‚ 1.90283203125 disk.nvme0n1.in‚ 670.1484375 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17% disk./.usageGB‚1053.2489585876465 disk.rbd0.out‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 378.2578125 disk.dm-0.in‚ 670.1484375" disk.dm-0.out‚50755.9150390625 disk.sda3.in‚ 0.07421875 cpu‚25.28534803810455! disk.nvme0c0n1.in‚ 670.1484375 disk.dm-1.in‚ 0.07421875% disk.nvme0n1.out‚50755.9150390625 disk.sda.in‚ 0.07421875½ Y¢2˜- _wandbruntime‚4050  _runtime‚4050Ûðəà ' lr-AdamW/pg1‚0.00019541801880766456 ' lr-AdamW/pg2‚0.00019541801880766456 trainer/global_step‚3699 % _timestamp‚1.7883989871322005e+09 _runtime‚4050.692051673 _step‚301®_BºÎšÈ _step‚301 ' lr-AdamW/pg1‚0.00019541801880766456 ' lr-AdamW/pg2‚0.00019541801880766456 trainer/global_step‚3699 % _timestamp‚1.7883989871322005e+09 _runtime‚4050.692051673­É~Àè2›- _wandbruntime‚4050  _runtime‚4050Ê×qìÜ'œÖO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 trainer/global_step‚3699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚149267924892672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 _runtime‚4050.693049974 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67840730 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134040617245696 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6738778318514 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98324190 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70170243 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41161786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6835357429248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134040634285056 _step‚302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168468249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142508973937152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293670912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165819107408896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70170241 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6759184541184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142508973937152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70170241 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168494433 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚728 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98298008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6759184541184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165819131348992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚149268176481280 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6758950955520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67841458 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172654195107328 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚149267924892672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6758950955520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26679672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6835087698432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172654488778240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 % _timestamp‚1.7883989871329658e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚149268176481280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504)‚Íá'ÛO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98298008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269730816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142508991940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41161784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚140779166594358 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚149268176481280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 trainer/global_step‚3699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚149267924892672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 _runtime‚4050.693049974 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67840730 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319570432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134040617245696 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6738778318514 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98324190 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70170243 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41161786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6835357429248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134040634285056 _step‚302 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168468249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142508973937152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293670912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165819107408896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70170241 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6759184541184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142508973937152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70170241 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168494433 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚728 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98298008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6759184541184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165819131348992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚149268176481280 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6758950955520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67841458 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172654195107328 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚149267924892672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6758950955520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26679672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6835087698432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172654488778240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 % _timestamp‚1.7883989871329658e+09 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70188422 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 29ìÏè33598976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134075543939072 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142546116227072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70188420 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚692 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _runtime‚4051.585781604 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168538713 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149306876331008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142546134230016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67859307 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67858613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149307127932928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273911808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6836927692800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168538713 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41172439 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 % _timestamp‚1.7883989880252182e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98350291 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41172437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26686176 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168512503 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70188420 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98350291 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98324083 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149307127932928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 trainer/global_step‚3699 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚140815897021174 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6760760103936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6740353082102 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6760993702912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0<º×hë& åM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98324083 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134075560978432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 297851904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70188422 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134075543939072 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142546116227072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70188420 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚692 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _runtime‚4051.585781604 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168538713 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149306876331008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142546134230016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67859307 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67858613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149307127932928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 273911808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6836927692800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168538713 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41172439 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 % _timestamp‚1.7883989880252182e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98350291 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41172437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26686176 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168512503 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70188420 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98350291 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98324083 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149307127932928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 trainer/global_step‚3699 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚140815897021174 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6760760103936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6740353082102 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6760993702912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¯¦"7ì2¡- _wandbruntime‚4051  _runtime‚4051Ô~Q¢Ë 1loss/SRqqt2/assignment_loss‚2.474170857532458 0loss/FBt2/assignment_loss‚0.7922983853347562 /loss/FRt3/detection_loss‚2.7707649832873633 /loss/FBt2/detection_loss‚3.1196095517703464 1loss/SRqqt1/assignment_loss‚2.515367695263454 1loss/SRqqt4/assignment_loss‚1.966324156478924 % _timestamp‚1.7883989880270612e+09 _runtime‚4051.586784493 .loss/FRt4/detection_loss‚1.967801276959625 /loss/FBt4/detection_loss‚0.8592366163895743 trainer/global_step‚3699 _step‚304 0loss/FRt2/assignment_loss‚3.9925590496437224 0loss/FBt1/assignment_loss‚0.5822011893884607 1loss/SRqqt1/detection_loss‚2.9930153006599065 epoch‚3 .loss/FRt2/detection_loss‚2.242614750184265 /loss/FBt1/detection_loss‚1.9186740062303014 .loss/FBt3/detection_loss‚2.810756340622902 $loss/total_loss‚55.9203459704208 /loss/FRt3/assignment_loss‚5.333316297242136 .loss/FRt1/detection_loss‚2.621904315317378 0loss/FRt4/assignment_loss‚3.0171601538564645 0loss/FBt3/assignment_loss‚0.4996168343350291 1loss/FBt4/assignment_loss‚0.29544992274451476 /loss/SRqqt3/detection_loss‚2.53301152677247 0loss/SRqqt4/detection_loss‚1.103058378047803 /loss/FRt1/assignment_loss‚5.238090795629165 0loss/SRqqt3/assignment_loss‚2.92371803567265 1loss/SRqqt2/detection_loss‚1.3496255510571329F¬C4V£Ð .loss/FRt4/detection_loss‚1.967801276959625 /loss/FBt4/detection_loss‚0.8592366163895743 trainer/global_step‚3699 _step‚304 0loss/FRt2/assignment_loss‚3.9925590496437224 0loss/FBt1/assignment_loss‚0.5822011893884607 1loss/SRqqt1/detection_loss‚2.9930153006599065 epoch‚3 .loss/FRt2/detection_loss‚2.242614750184265 /loss/FBt1/detection_loss‚1.9186740062303014 .loss/FBt3/detection_loss‚2.810756340622902 $loss/total_loss‚55.9203459704208 /loss/FRt3/assignment_loss‚5.333316297242136 .loss/FRt1/detection_loss‚2.621904315317378 0loss/FRt4/assignment_loss‚3.0171601538564645 0loss/FBt3/assignment_loss‚0.4996168343350291 1loss/FBt4/assignment_loss‚0.29544992274451476 /loss/SRqqt3/detection_loss‚2.53301152677247 0loss/SRqqt4/detection_loss‚1.103058378047803 /loss/FRt1/assignment_loss‚5.238090795629165 0loss/SRqqt3/assignment_loss‚2.92371803567265 1loss/SRqqt2/detection_loss‚1.3496255510571329 1loss/SRqqt2/assignment_loss‚2.474170857532458 0loss/FBt2/assignment_loss‚0.7922983853347562 /loss/FRt3/detection_loss‚2.7707649832873633 /loss/FBt2/detection_loss‚3.1196095517703464 1loss/SRqqt1/assignment_loss‚2.515367695263454 1loss/SRqqt4/assignment_loss‚1.966324156478924 % _timestamp‚1.7883989880270612e+09 _runtime‚4051.586784493°e”_‘¤‚(:† ™›ãÔ炶Ó gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5226 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fSŽRA¥‚(:¶ ™›ãÔ¢‡¢Õ disk.rbd3.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2846.33984375% disk./.usageGB‚1053.4991188049316 disk.drbd1777.out‚0% disk.nvme0c0n1.out‚50855.10546875 cpu‚25.284443034384 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 1.90283203125 disk.rbd0.out‚0 disk.sda.out‚ 3082.69921875 network.recv‚ 1.451018e+06 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 670.1484375 disk.sda.in‚ 0.07421875 network.sent‚ 8.723612e+06 disk.sda3.out‚ 3082.69921875' proc.memory.availableMB‚ 287.2265625* proc.memory.percent‚17.372679710388184 disk.rbd2.out‚ 126.87890625 disk.dm-0.out‚50855.10546875$ memory_percent‚98.24690818786621 proc.cpu.threads‚17) disk./.usagePercent‚70.70120980763635 disk.drbd1777.in‚0 disk.dm-0.in‚ 670.1484375 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 670.1484375 disk.dm-1.out‚ 3082.69921875 disk.rbd0.in‚0# disk.nvme0n1.out‚50855.10546875‹uÈ5œ¦‚(:‘ ¨›ãÔ•€ðÒ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.854' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.570800000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C1kE§‚(:º ¨›ãÔúÏÑÕ disk.drbd1777.out‚0% disk.nvme0n1.out‚50855.8134765625 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.sda3.out‚ 3083.30859375 disk.rbd1.out‚ 1.90283203125! disk.nvme0c0n1.in‚ 670.1484375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.1484375 disk.sda.out‚ 3083.30859375" disk./.usageGB‚1053.4990234375 disk.rbd3.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚50855.8134765625 disk.drbd1777.in‚0 disk.rbd2.out‚ 127.00390625 disk.dm-1.out‚ 3083.30859375& proc.memory.availableMB‚ 198.921875# proc.memory.rssMB‚ 2846.33984375 disk.dm-1.in‚ 0.07421875" disk.dm-0.out‚50855.8134765625 cpu‚25.283581911581866 disk.dm-0.in‚ 670.1484375 disk.sda2.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.372679710388184 network.sent‚ 8.832686e+06) disk./.usagePercent‚70.70120340744799 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0# memory_percent‚98.7858772277832 network.recv‚ 1.457657e+06 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0øTó¨‚(:… ·›ãÔÔ¸’Ó gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.385' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.277 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÙ™B©‚(:· ·›ãÔ¾š¦Õ! disk.nvme0c0n1.in‚ 670.1484375% disk./.usageGB‚1053.4990196228027 disk.sda.out‚ 3083.6484375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90283203125" disk.dm-0.out‚50856.1337890625$ memory_percent‚98.92749786376953# proc.memory.rssMB‚ 2846.33984375 disk.sda.in‚ 0.07421875 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 670.1484375 disk.drbd1777.in‚0% disk.nvme0n1.out‚50856.1337890625 disk.dm-0.in‚ 670.1484375 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd2.out‚ 127.01953125 disk.sda2.in‚0 disk.rbd0.in‚0 cpu‚25.2826755688274 network.sent‚ 8.837834e+06 disk.sda3.out‚ 3083.6484375 network.recv‚ 1.46136e+06 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.372679710388184 disk.rbd3.in‚0' disk.nvme0c0n1.out‚50856.1337890625 disk.dm-1.out‚ 3083.6484375 disk.rbd0.out‚0% proc.memory.availableMB‚ 175.71875 proc.cpu.threads‚17) disk./.usagePercent‚70.70120315144047íB·ñ2ª- _wandbruntime‚4097  _runtime‚4097ÒÛ½àÉ«Ã _runtime‚4097.659506896 _step‚305 ' lr-AdamW/pg1‚0.00019434918822644492 ' lr-AdamW/pg2‚0.00019434918822644492 trainer/global_step‚3749 % _timestamp‚1.7883990340996974e+09¶´”™Î¬È ' lr-AdamW/pg1‚0.00019434918822644492 ' lr-AdamW/pg2‚0.00019434918822644492 trainer/global_step‚3749 % _timestamp‚1.7883990340996974e+09 _runtime‚4097.659506896 _step‚305±ƒÝë2­- _wandbruntime‚4097  _runtime‚4097AKÆd®ÖO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135786951914496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295768064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174914836887040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7883990341004338e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71079191 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271827968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142615947535578 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚151215511223296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170710485 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144366088433152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151215511223296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99631292 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99631292 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71079193 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6849422790144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6828978581722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68717975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144366088433152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6849656375808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚151215762811904 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚738 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6926760053248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142615701526366 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960­À¯ á'¯ÛO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41694436 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71079193 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6849422790144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6828978581722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68717975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144366088433152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6849656375808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚151215762811904 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚738 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6926760053248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142615701526366 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135786951914496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295768064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174914836887040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7883990341004338e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71079191 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271827968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142615947535578 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚151215511223296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170710485 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144366088433152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151215511223296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99631292 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99631292 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144403248726016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168031431803392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6851465537536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71097372 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27030439 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142652677962602 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3749 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170754765 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68735528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135821895647232 _step‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99657393 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99657393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135821878607872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170728555 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7883990349978976e+09 _runtime‚4098.558418307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71097370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144403230723072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6830553345310 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41705087 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27029746 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6851231938560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6830782315370 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6928313757696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142652431953182 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚151254714263552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68734833 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144403230723072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99631185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99631185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560ÉTOl#²åM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142652677962602 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3749 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170754765 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68735528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135821895647232 _step‚307 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99657393 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99657393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135821878607872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170728555 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7883990349978976e+09 _runtime‚4098.558418307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71097370 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144403230723072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6830553345310 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41705087 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27029746 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6851231938560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6830782315370 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6928313757696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142652431953182 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚151254714263552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68734833 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144403230723072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99631185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99631185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41705089 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6928581378048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144403248726016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168031431803392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319570432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6851465537536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71097372 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27030439 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0³-±—«2³- _wandbruntime‚4098  _runtime‚4098?12rR´Ì 2loss/SRqqt1/assignment_loss‚3.0072906517982485 /loss/FBt2/assignment_loss‚0.849183764308691 trainer/global_step‚3749 /loss/FRt2/detection_loss‚1.9679816232787237 /loss/FRt4/detection_loss‚1.8576178740636975 /loss/FRt2/assignment_loss‚3.455387738015917 /loss/FRt3/assignment_loss‚5.954698173205058 0loss/FBt4/assignment_loss‚0.2382218715456966 .loss/FRt3/detection_loss‚2.910729823509852 1loss/SRqqt4/detection_loss‚1.1710389740765095 /loss/FBt4/detection_loss‚0.8594552226163246 /loss/FRt1/assignment_loss‚5.043685396512349 /loss/FRt4/assignment_loss‚2.582130501083299 2loss/SRqqt4/assignment_loss‚2.0162995776627213 0loss/SRqqt1/detection_loss‚3.144710102081299 0loss/SRqqt3/detection_loss‚2.601109631359577 1loss/SRqqt3/assignment_loss‚2.616266008466482 epoch‚3 $ _timestamp‚1.788399034999833e+09 /loss/FBt1/assignment_loss‚0.548169376826263 .loss/FBt2/detection_loss‚2.429504372179508 .loss/FBt3/detection_loss‚2.788742780685425 _runtime‚4098.559544248 0loss/FBt3/assignment_loss‚0.6302557867925821 /loss/FRt1/detection_loss‚2.6517336666584015 1loss/SRqqt2/detection_loss‚1.4357815030962229 %loss/total_loss‚54.88660219023042 2loss/SRqqt2/assignment_loss‚2.1911880165338515 /loss/FBt1/detection_loss‚1.9354197538737208 _step‚308F¬ÊWµÑ epoch‚3 $ _timestamp‚1.788399034999833e+09 /loss/FBt1/assignment_loss‚0.548169376826263 .loss/FBt2/detection_loss‚2.429504372179508 .loss/FBt3/detection_loss‚2.788742780685425 _runtime‚4098.559544248 0loss/FBt3/assignment_loss‚0.6302557867925821 /loss/FRt1/detection_loss‚2.6517336666584015 1loss/SRqqt2/detection_loss‚1.4357815030962229 %loss/total_loss‚54.88660219023042 2loss/SRqqt2/assignment_loss‚2.1911880165338515 /loss/FBt1/detection_loss‚1.9354197538737208 _step‚308 2loss/SRqqt1/assignment_loss‚3.0072906517982485 /loss/FBt2/assignment_loss‚0.849183764308691 trainer/global_step‚3749 /loss/FRt2/detection_loss‚1.9679816232787237 /loss/FRt4/detection_loss‚1.8576178740636975 /loss/FRt2/assignment_loss‚3.455387738015917 /loss/FRt3/assignment_loss‚5.954698173205058 0loss/FBt4/assignment_loss‚0.2382218715456966 .loss/FRt3/detection_loss‚2.910729823509852 1loss/SRqqt4/detection_loss‚1.1710389740765095 /loss/FBt4/detection_loss‚0.8594552226163246 /loss/FRt1/assignment_loss‚5.043685396512349 /loss/FRt4/assignment_loss‚2.582130501083299 2loss/SRqqt4/assignment_loss‚2.0162995776627213 0loss/SRqqt1/detection_loss‚3.144710102081299 0loss/SRqqt3/detection_loss‚2.601109631359577 1loss/SRqqt3/assignment_loss‚2.616266008466482´!Ƕ‚(:… Æ›ãÔ‚Ä„Ô gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9084 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ç¡ÅH·‚(:½ Æ›ãÔê­…Ö disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 3084.328125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 670.1484375 disk.sda.out‚ 3084.328125 disk.drbd1777.out‚0 cpu‚25.281822267000617 disk.sda3.out‚ 3084.328125 network.recv‚ 1.465896e+06 disk.rbd3.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚50955.43115234375% disk./.usageGB‚1053.4994583129883 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.sda1.in‚0* proc.memory.percent‚17.372679710388184 disk.rbd1.out‚ 1.90283203125 network.sent‚ 8.844149e+06 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 670.1484375& disk.nvme0n1.out‚50955.43115234375 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 670.1484375& proc.memory.availableMB‚ 175.890625# proc.memory.rssMB‚ 2846.33984375 disk.sda.in‚ 0.07421875( disk./.usagePercent‚70.7012325923069! disk.rbd2.out‚127.10595703125( disk.nvme0c0n1.out‚50955.43115234375$ memory_percent‚98.92644882202148!b¸‚(:„ Õ›ãÔÊùüÒ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.4972 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÔo¤B¹‚(:· Õ›ãÔ¸—ÄÖ disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 3084.6875 disk.rbd3.out‚0 disk.sda3.out‚ 3084.6875 disk.dm-1.in‚ 0.07421875 disk.sda.in‚ 0.07421875 cpu‚25.28103882118181 disk.dm-0.in‚ 670.1484375) disk./.usagePercent‚70.70124718473635 disk.sda1.out‚0 disk.rbd1.out‚ 2.05126953125( disk.nvme0c0n1.out‚50955.70849609375 disk.rbd0.out‚0& proc.memory.availableMB‚ 175.640625 disk.drbd1777.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚127.11767578125 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875! disk.nvme0c0n1.in‚ 670.1484375# proc.memory.rssMB‚ 2846.33984375# disk.dm-0.out‚50955.70849609375 network.recv‚ 1.471826e+06% disk./.usageGB‚1053.4996757507324 disk.sda2.in‚0 disk.nvme0n1.in‚ 670.1484375& disk.nvme0n1.out‚50955.70849609375 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 3084.6875$ memory_percent‚98.92797470092773 proc.cpu.threads‚17 network.sent‚ 8.952248e+06 disk.rbd0.in‚0* proc.memory.percent‚17.372679710388184grYÀº‚(:… ä›ãÔž¤ìÒ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.565' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.113 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00/ç½K»‚(:À ä›ãÔ±““Õ network.recv‚ 1.476407e+06( disk.nvme0c0n1.out‚50956.25244140625 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚128.75439453125 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚50956.25244140625# proc.memory.rssMB‚ 2846.33984375 proc.cpu.threads‚17 disk.dm-0.in‚ 670.1484375# memory_percent‚98.9189624786377* proc.memory.percent‚17.372679710388184 cpu‚25.28016444291848) disk./.usagePercent‚70.70123156827675 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 2.05126953125& disk.nvme0n1.out‚50956.25244140625 disk.drbd1777.out‚0 disk.sda1.out‚0% disk./.usageGB‚1053.4994430541992 disk.sda3.out‚ 3085.3046875! disk.nvme0c0n1.in‚ 670.1484375 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 3085.3046875 disk.sda.in‚ 0.07421875 network.sent‚ 8.958647e+06 disk.sda.out‚ 3085.3046875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 670.1484375' proc.memory.availableMB‚ 177.1171875 disk.rbd3.out‚0 disk.sda2.out‚0á’Á2¼- _wandbruntime‚4144  _runtime‚4144óõ¿lǽÁ % _timestamp‚1.7883990810266037e+09 _runtime‚4144.586396366 _step‚309 & lr-AdamW/pg1‚0.0001932630934620277 & lr-AdamW/pg2‚0.0001932630934620277 trainer/global_step‚3799› §Ì¾Æ _runtime‚4144.586396366 _step‚309 & lr-AdamW/pg1‚0.0001932630934620277 & lr-AdamW/pg2‚0.0001932630934620277 trainer/global_step‚3799 % _timestamp‚1.7883990810266037e+09µ•å–‹2¿- _wandbruntime‚4144  _runtime‚4144‰kÉà'ÀÚO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100938394 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27367686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71988141 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177174852771328 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170157022318592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146223202929152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153163097553920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6940128210432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177175135956480 _runtime‚4144.587348159 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146223202929152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6939894624768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7017854392832 trainer/global_step‚3799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71988141 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 _step‚310 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100912212 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6940128210432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153163349142528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42227086 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172900353 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137533286583296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42227084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153163097553920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144452236458374 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6940128210432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153163349142528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42227086 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172900353 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137533286583296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42227084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153163097553920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144452236458374 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146223202929152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153163097553920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6940128210432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177175135956480 _runtime‚4144.587348159 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146223202929152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6939894624768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7017854392832 trainer/global_step‚3799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71988141 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 _step‚310 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560¶6gš2Â- _wandbruntime‚4145  _runtime‚4145­~F{¶ ÃäM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172970817 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170200401228288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170200377288192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100938287 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72006320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146260363222016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 _step‚311 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523200 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7019682572288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289463296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _runtime‚4145.470480306 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172970817 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool>3¨4.current‚26208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27373876 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153202300594176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 % _timestamp‚1.7883990819098365e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69612309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172944607 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6941937372160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42237739 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7019948095488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144488966885190 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6941703773184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146260345219072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146260363222016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42237737 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153202048992256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27374570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177220059860480 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144489212894610 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137568213276672 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69611613 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6920982578578 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 trainer/global_step‚3799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72006320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172944607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137568230316032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177220349323776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6941703773184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72006322 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153202300594176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27373876 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153202300594176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 % _timestamp‚1.7883990819098365e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69612309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172944607 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6941937372160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42237739 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7019948095488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144488966885190 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6941703773184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146260345219072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146260363222016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42237737 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153202048992256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27374570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177220059860480 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144489212894610 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137568213276672 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69611613 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6920982578578 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 trainer/global_step‚3799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72006320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172944607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚137568230316032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177220349323776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6941703773184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72006322 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153202300594176 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172970817 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170200401228288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170200377288192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100938287 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72006320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146260363222016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336·Í €2Å- _wandbruntime‚4145  _runtime‚4145ë"¹CUÆÏ .loss/FRt1/detection_loss‚2.584759571973015 /loss/FRt1/assignment_loss‚5.515999429366168 0loss/FRt4/assignment_loss‚3.0384790437263356 0loss/FBt1/assignment_loss‚0.5421793100078879 1loss/FBt2/assignment_loss‚0.28233304794412106 1loss/FBt4/assignment_loss‚0.27216879105799974 /loss/FBt3/detection_loss‚2.4382147043943405 _runtime‚4145.471674156 /loss/FRt2/assignment_loss‚4.052100295923194 .loss/FBt2/detection_loss‚3.167355239391327 #loss/total_loss‚54.622780660092 _step‚312 /loss/FRt2/detection_loss‚2.3201379605701993 /loss/FRt4/detection_loss‚1.7975716000063378 trainer/global_step‚3799 0loss/FBt3/assignment_loss‚0.8317627544874995 /loss/FRt3/detection_loss‚2.5989910027255183 1loss/SRqqt1/detection_loss‚2.4323878064751625 0loss/SRqqt3/detection_loss‚2.376236606727947 /loss/FBt4/detection_loss‚0.7975153975079164 0loss/SRqqt4/detection_loss‚1.319076019720953 2loss/SRqqt3/assignment_loss‚3.0671950723185684 0loss/FRt3/assignment_loss‚4.6657133983529135 2loss/SRqqt1/assignment_loss‚2.9661936536431313 2loss/SRqqt4/assignment_loss‚1.9243496386486976 0loss/SRqqt2/detection_loss‚1.507749396096915 /loss/FBt1/detection_loss‚1.6994195358249649 epoch‚3 1loss/SRqqt2/assignment_loss‚2.424891383200884 # _timestamp‚1.78839908191192e+09´ŒaZÇÔ 0loss/FRt3/assignment_loss‚4.6657133983529135 2loss/SRqqt1/assignment_loss‚2.9661936536431313 2loss/SRqqt4/assignment_loss‚1.9243496386486976 0loss/SRqqt2/detection_loss‚1.507749396096915 /loss/FBt1/detection_loss‚1.6994195358249649 epoch‚3 1loss/SRqqt2/assignment_loss‚2.424891383200884 # _timestamp‚1.78839908191192e+09 .loss/FRt1/detection_loss‚2.584759571973015 /loss/FRt1/assignment_loss‚5.515999429366168 0loss/FRt4/assignment_loss‚3.0384790437263356 0loss/FBt1/assignment_loss‚0.5421793100078879 1loss/FBt2/assignment_loss‚0.28233304794412106 1loss/FBt4/assignment_loss‚0.27216879105799974 /loss/FBt3/detection_loss‚2.4382147043943405 _runtime‚4145.471674156 /loss/FRt2/assignment_loss‚4.052100295923194 .loss/FBt2/detection_loss‚3.167355239391327 #loss/total_loss‚54.622780660092 _step‚312 /loss/FRt2/detection_loss‚2.3201379605701993 /loss/FRt4/detection_loss‚1.7975716000063378 trainer/global_step‚3799 0loss/FBt3/assignment_loss‚0.8317627544874995 /loss/FRt3/detection_loss‚2.5989910027255183 1loss/SRqqt1/detection_loss‚2.4323878064751625 0loss/SRqqt3/detection_loss‚2.376236606727947 /loss/FBt4/detection_loss‚0.7975153975079164 0loss/SRqqt4/detection_loss‚1.319076019720953 2loss/SRqqt3/assignment_loss‚3.0671950723185684¸Æ È‚(:… ó›ãÔüÊëÓ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.085' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.417 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦nHÉ‚(:½ ó›ãÔ™³ÊÕ disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚51055.34228515625 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda3.out‚ 3085.828125 disk.dm-0.in‚ 670.1484375( disk.nvme0c0n1.out‚51055.34228515625 disk.dm-1.in‚ 0.07421875# memory_percent‚98.9161491394043 proc.cpu.threads‚17 disk.nvme0n1.in‚ 670.1484375 network.sent‚ 8.963854e+06 network.recv‚ 1.480087e+06 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 2.08349609375 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚70.70123668842744% disk./.usageGB‚1053.4995193481445 disk.rbd2.in‚0! disk.rbd2.out‚128.75439453125! disk.nvme0c0n1.in‚ 670.1484375 disk.dm-1.out‚ 3085.828125 disk.sda.in‚ 0.07421875* proc.memory.percent‚17.372679710388184 cpu‚25.279333316347397 disk.rbd3.in‚0& disk.nvme0n1.out‚51055.34228515625& proc.memory.availableMB‚ 177.578125# proc.memory.rssMB‚ 2846.33984375 disk.sda.out‚ 3085.8281252ª"‘Ê‚(:† ‚œãÔš±êÒ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1708 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ‹›MË‚(: ‚œãÔñÃÕ% disk./.usageGB‚1053.4815864562988! disk.rbd2.out‚128.83251953125 disk.sda2.out‚0 disk.nvme0n1.in‚ 670.16796875' proc.memory.availableMB‚ 177.6484375# proc.memory.rssMB‚ 2846.33984375 network.sent‚ 9.072683e+06 network.recv‚ 1.486518e+06 disk.sda.out‚ 3086.35546875 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 670.16796875! disk.dm-0.out‚51060.876953125 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚70.70003319700898 disk.dm-1.out‚ 3086.35546875 disk.sda2.in‚0& disk.nvme0c0n1.out‚51060.876953125 disk.rbd0.out‚0$ memory_percent‚98.91571998596191 cpu‚25.278520788453264 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.dm-1.in‚ 0.07421875* proc.memory.percent‚17.372679710388184 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.08349609375$ disk.nvme0n1.out‚51060.876953125 proc.cpu.threads‚17 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3086.35546875 disk.rbd0.in‚0¤“ëž‘Ì‚(:† ‘œãÔÁ™‡Ó gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.742' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9484 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cßFÍ‚(:» ‘œãÔئÒÕ! disk.dm-0.out‚51159.978515625 disk.drbd1777.in‚0 disk.dm-0.in‚ 670.16796875 disk.sda.in‚ 0.07421875* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 177.890625 cpu‚25.277698996423602 network.sent‚ 9.078104e+06 disk.rbd2.in‚0 disk.rbd0.in‚0$ memory_percent‚98.91424179077148# proc.memory.rssMB‚ 2846.33984375 disk.sda1.in‚0$ disk.nvme0n1.out‚51159.978515625 disk.sda2.out‚0) disk./.usagePercent‚70.71681193080309! disk.rbd2.out‚128.83251953125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3086.7578125 disk.rbd1.out‚ 2.146484375$ disk./.usageGB‚1053.731601715088 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 670.16796875 disk.sda.out‚ 3086.7578125 network.recv‚ 1.490511e+06 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 670.16796875& disk.nvme0c0n1.out‚51159.978515625 disk.dm-1.out‚ 3086.7578125 disk.rbd0.out‚0ùŒ?2Î- _wandbruntime‚4191  _runtime‚4191ߺvLÉÏà _step‚313 ' lr-AdamW/pg1‚0.00019215996085493404 ' lr-AdamW/pg2‚0.00019215996085493404 trainer/global_step‚3849 % _timestamp‚1.7883991279531512e+09 _runtime‚4191.512986727¬”òÎÐÈ % _timestamp‚1.7883991279531512e+09 _runtime‚4191.512986727 _step‚313 ' lr-AdamW/pg1‚0.00019215996085493404 ' lr-AdamW/pg2‚0.00019215996085493404 trainer/global_step‚3849¹aw¾2Ñ- _wandbruntime‚4191  _runtime‚41917>~!ÒÙO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚737 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179435446374400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7109478571008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚155110683884544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102219314 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267633664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72897091 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102219314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155110683884544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚148080317425152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7883991279538865e+09 _step‚314 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175116405 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72897093 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72897091 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42759734 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72897093 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7009379108138 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 trainer/global_step‚3849 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7030600045056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102245496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291573760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7030600045056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146288771390382 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155110935473152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7030366459392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148080335428096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7109210937344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139279621252096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚739 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 UGDeviceStatsMonit¶îªXÛor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175142589 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70471526 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42759736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139279638291456 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42759734 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72897093 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7009379108138 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 trainer/global_step‚3849 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7030600045056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102245496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291573760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7030600045056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146288771390382 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155110935473152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7030366459392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚148080335428096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7109210937344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139279621252096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚739 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175142589 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70471526 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42759736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139279638291456 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102219314 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267633664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72897091 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102219314 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155110683884544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚148080317425152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7883991279538865e+09 _step‚314 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175116405 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72897093ºòµõ2Ô- _wandbruntime‚4192  _runtime‚4192AG¡ê&ÕäM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 _runtime‚4192.409283762 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175186869 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 _step‚315 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7032175607808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72915272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7010953871726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42770389 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285268992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42770387 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146325501817198 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102271597 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148117477718016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148117459715072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27718460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139314564984832 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179480646913024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 trainer/global_step‚3849 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172369346713088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7032175607808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155149635322880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102271597 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72915270 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139314547945472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70488847 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172369322772992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 % _timestamp‚1.7883991288489046e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7032409206784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179480361644032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146325747826618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175186869 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72915270 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72915272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102245389ÒÌøJï&ÖéM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42770389 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285268992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42770387 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146325501817198 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102271597 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148117477718016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148117459715072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27718460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139314564984832 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179480646913024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 trainer/global_step‚3849 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172369346713088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7032175607808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155149635322880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102271597 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72915270 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139314547945472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70488847 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172369322772992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 % _timestamp‚1.7883991288489046e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7032409206784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179480361644032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146325747826618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175186869 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72915270 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚706 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72915272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102245389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 _runtime‚4192.409283762 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175186869 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 _step‚315 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7032175607808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72915272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7010953871726»ù…,¿2×- _wandbruntime‚4192  _runtime‚4192_-R'ÓØÑ /loss/FBt1/detection_loss‚2.4860179296160827 /loss/FBt2/detection_loss‚2.9539289474487305 /loss/FRt2/assignment_loss‚4.245239306003489 2loss/SRqqt1/assignment_loss‚2.8345660737582614 1loss/SRqqt2/assignment_loss‚2.290751487424929 1loss/SRqqt3/detection_loss‚2.5546903548569513 /loss/FBt4/detection_loss‚0.8190317688534012 % _timestamp‚1.7883991288507178e+09 0loss/SRqqt1/detection_loss‚2.556365858231272 .loss/FBt3/detection_loss‚3.006496979130639 _step‚316 0loss/FBt1/assignment_loss‚0.5572932374175602 0loss/FBt2/assignment_loss‚0.2375186262652278 1loss/FBt4/assignment_loss‚0.24465778485243875 %loss/total_loss‚54.93981231640898 0loss/FRt4/assignment_Š¤¿d„loss‚2.6446394843083842 /loss/FRt1/assignment_loss‚5.442409730619854 .loss/FRt1/detection_loss‚2.696612815062205 /loss/FRt3/detection_loss‚2.4576994336464186 /loss/FRt3/assignment_loss‚5.102600119330666 2loss/SRqqt4/assignment_loss‚1.9996164185342504 /loss/FRt4/detection_loss‚1.9300682715795658 1loss/SRqqt2/detection_loss‚1.5917591036182561 epoch‚3 _runtime‚4192.410416847 2loss/SRqqt3/assignment_loss‚2.3250924410491156 trainer/global_step‚3849 0loss/FBt3/assignment_loss‚0.6418276735182897 .loss/FRt2/detection_loss‚2.298577526148329 0loss/SRqqt4/detection_loss‚1.022350945134661&íq\ÙÖ 0loss/SRqqt1/detection_loss‚2.556365858231272 .loss/FBt3/detection_loss‚3.006496979130639 _step‚316 0loss/FBt1/assignment_loss‚0.5572932374175602 0loss/FBt2/assignment_loss‚0.2375186262652278 1loss/FBt4/assignment_loss‚0.24465778485243875 %loss/total_loss‚54.93981231640898 0loss/FRt4/assignment_loss‚2.6446394843083842 /loss/FRt1/assignment_loss‚5.442409730619854 .loss/FRt1/detection_loss‚2.696612815062205 /loss/FRt3/detection_loss‚2.4576994336464186 /loss/FRt3/assignment_loss‚5.102600119330666 2loss/SRqqt4/assignment_loss‚1.9996164185342504 /loss/FRt4/detection_loss‚1.9300682715795658 1loss/SRqqt2/detection_loss‚1.5917591036182561 epoch‚3 _runtime‚4192.410416847 2loss/SRqqt3/assignment_loss‚2.3250924410491156 trainer/global_step‚3849 0loss/FBt3/assignment_loss‚0.6418276735182897 .loss/FRt2/detection_loss‚2.298577526148329 0loss/SRqqt4/detection_loss‚1.022350945134661 /loss/FBt1/detection_loss‚2.4860179296160827 /loss/FBt2/detection_loss‚2.9539289474487305 /loss/FRt2/assignment_loss‚4.245239306003489 2loss/SRqqt1/assignment_loss‚2.8345660737582614 1loss/SRqqt2/assignment_loss‚2.290751487424929 1loss/SRqqt3/detection_loss‚2.5546903548569513 /loss/FBt4/detection_loss‚0.8190317688534012 % _timestamp‚1.7883991288507178e+09¼vëFÚ‚(:„  œãÔãšÿÒ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚84.038' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.8076 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëòIÛ‚(:¾  œãÔˆÁ¾Õ network.sent‚ 9.0835e+06 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 670.16796875 disk.sda.in‚ 0.07421875$ memory_percent‚98.91767501831055# proc.memory.rssMB‚ 2846.33984375 network.recv‚ 1.494527e+06 disk.sda3.out‚ 3087.30859375 disk.rbd1.in‚0 disk.dm-1.out‚ 3087.30859375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.16796875) disk./.usagePercent‚70.71681628293118 disk.sda2.in‚0 disk.sda3.in‚ 0.07421875' disk.nvme0c0n1.out‚51160.5166015625% disk.nvme0n1.out‚51160.5166015625 disk.rbd1.out‚ 2.146484375" disk.dm-0.out‚51160.5166015625 disk.sda.out‚ 3087.30859375 disk.sda1.out‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17 disk.dm-1.in‚ 0.07421875 disk.sda1.in‚0 disk.drbd1777.out‚0& proc.memory.availableMB‚ 177.328125% disk./.usageGB‚1053.7316665649414" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd0.in‚0 cpu‚25.27689284455099 disk.rbd2.out‚128.9970703125"sìäŽÜ‚(:ƒ ¯œãÔʵÿÒ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.02' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.204 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì«LÝ‚(:Á ¯œãÔçºÒÖ disk.rbd3.out‚0 network.sent‚ 9.191137e+06 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 670.16796875 disk.sda1.in‚0 disk.rbd1.out‚ 2.20947265625# disk.dm-0.out‚51259.75927734375 disk.sda.in‚ 0.07421875 disk.sda.out‚ 3087.7421875& proc.memory.availableMB‚ 177.578125 proc.cpu.threads‚17 disk.sda2.in‚0% disk./.usageGB‚1053.7317276000977 disk.drbd1777.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd0.in‚0# memory_percent‚98.9161491394043* proc.memory.percent‚17.372679710388184) disk./.usagePercent‚70.71682037905171 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 3087.7421875# proc.memory.rssMB‚ 2846.33984375 disk.rbd2.out‚128.9970703125 disk.sda3.out‚ 3087.7421875( disk.nvme0c0n1.out‚51259.75927734375& disk.nvme0n1.out‚51259.75927734375 disk.sda2.out‚0 network.recv‚ 1.500135e+06 disk.sda1.out‚0 disk.nvme0n1.in‚ 670.16796875 cpu‚25.27607618838128¿;ø˜œÞ‚(:‘ ¾œãÔÚÙøÒ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.678' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.735599999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#‡O8Pß‚(:Å ¾œãÔ›µ¨Õ) disk./.usagePercent‚70.71681653893872 disk.sda2.in‚0( disk.nvme0c0n1.out‚51260.83544921875 disk.rbd0.in‚0 disk.sda.out‚ 3088.33984375( proc.memory.availableMB‚ 177.57421875 cpu‚25.2752959014628 proc.cpu.threads‚17 network.recv‚ 1.504417e+06 disk.drbd1777.in‚0 disk.rbd2.out‚130.7119140625 disk.rbd3.out‚0# disk.dm-0.out‚51260.83544921875 disk.dm-1.out‚ 3088.33984375 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 2.20947265625 network.sent‚ 9.197415e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 670.16796875 disk.rbd2.in‚0* proc.memory.percent‚17.372679710388184" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd3.in‚0# proc.memory.rssMB‚ 2846.33984375 disk.sda1.in‚0 disk.dm-1.in‚ 0.07421875% disk./.usageGB‚1053.7316703796387 disk.sda2.out‚0 disk.sda3.out‚ 3088.33984375& disk.nvme0n1.out‚51260.83544921875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 670.16796875 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.drbd1777.out‚0# memory_percent‚98.9161729812622|œÐ`2à- _wandbruntime‚4238  _runtime‚4238[É@Èá _step‚317 ' lr-AdamW/pg1‚0.00019104002029634578 ' lr-AdamW/pg2‚0.00019104002029634578 trainer/global_step‚3899 % _timestamp‚1.7883991748542068e+09 _runtime‚ 4238.41401474¿´ÝÍâÇ ' lr-AdamW/pg2‚0.00019104002029634578 trainer/global_step‚3899 % _timestamp‚1.7883991748542068e+09 _runtime‚ 4238.41401474 _step‚317 ' lr-AdamW/pg1‚0.00019104002029634578½ëI*2ã- _wandbruntime‚4238  _runtime‚4238ßµà'äÚO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28059243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174494913288192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71352334 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141025972960256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7099350401494 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 % _timestamp‚1.7883991748549225e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43292384 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103552598 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149937431921152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7200637581312 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177332457 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43292386 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181695820600320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7121071879680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148125552331602 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7120838294016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73806041 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7121071879680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103526416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 trainer/global_step‚3899 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73806041 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149937431921152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71351627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7200907312128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177358641 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148125306322390 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103526416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157058270215168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149937449924096 _step‚318 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157058521803776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181695526929408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7099579371346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157058521803776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174494889348096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103552598 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73806043 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73806043 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157058270215168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28059948¶sçå'åßO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7121071879680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103526416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 trainer/global_step‚3899 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73806041 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149937431921152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71351627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7200907312128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177358641 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148125306322390 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103526416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157058270215168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149937449924096 _step‚318 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157058521803776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181695526929408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7099579371346 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚157058521803776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174494889348096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103552598 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73806043 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73806043 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157058270215168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28059948 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28059243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174494913288192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71352334 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141025972960256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7099350401494 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 % _timestamp‚1.7883991748549225e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43292384 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103552598 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149937431921152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7200637581312 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177332457 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43292386 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181695820600320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7121071879680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148125552331602 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7120838294016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73806041 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376¾Dy‚ç2æ- _wandbruntime‚4239  _runtime‚4239ߌ"çäM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103578699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43303037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177376711 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148162036749206 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73824220 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71368880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103552491 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141060882614272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7101154134934 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7122881041408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149974592214016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149974574211072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7122647442432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73824220 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚îÖ„È 251601920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181740746253824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚319 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73824222 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177402921 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177402921 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43303039 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174538292197888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7202745616384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7101383104994 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 trainer/global_step‚3899 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149974592214016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157097221653504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181741037814272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚713 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174538268257792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103552491 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177376711 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148162036749206 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73824220 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71368880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103552491 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141060882614272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7101154134934 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7122881041408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149974592214016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149974574211072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7122647442432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73824220 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181740746253824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 _step‚319 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73824222 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177402921 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177402921 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43303039 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174538292197888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7202745616384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7101383104994 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 trainer/global_step‚3899 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149974592214016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157097221653504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181741037814272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚713 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174538268257792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103552491 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7292380366336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142772307629056 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚322 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7211310128640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43825034 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104859700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183956239139328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159005856545792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28405599 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43825036 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104859700 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7292119024128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3949 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159006108134400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104833518 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149961841254398 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74714993 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚729 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72230633 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159005856545792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176663834832896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7211543714304 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104833518 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.al÷·›xlocated‚ 179574693 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179548509 _runtime‚4285.464413857Úîúpä'÷ÞO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7189779634554 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788399221904429e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7292380366336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142772307629056 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚322 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7211310128640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43825034 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104859700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183956239139328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159005856545792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28405599 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43825036 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104859700 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7292119024128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚3949 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159006108134400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104833518 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149961841254398 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74714993 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚729 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72230633 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159005856545792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176663834832896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7211543714304 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104833518 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179574693 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179548509 _runtime‚4285.464413857 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74714991 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74714991 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142772290589696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7211310128640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151794564420096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151794546417152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7211543714304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7189550664702 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72231362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74714993 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159006108134400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151794564420096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28406326 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176663858772992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3949 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142807217283072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179618973 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142807234322432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176707213742592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151831688707072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159045059586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74733172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 _runtime‚4286.363033531 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28413400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184001155867136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149998571681214 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43835689 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74733172 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚727 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7191583368202 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 % _timestamp‚1.7883992228027217e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179618973 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7294209744896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74733170 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7293942124544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72249089 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104885801 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 _step‚323 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚729 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72248360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Ñk_Õï&úéM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151831688707072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159045059586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74733172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 _runtime‚4286.363033531 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28413400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184001155867136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149998571681214 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43835689 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74733172 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚727 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7191583368202 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 % _timestamp‚1.7883992228027217e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179618973 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7294209744896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74733170 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7293942124544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72249089 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104885801 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 _step‚323 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚729 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72248360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151831706710016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7213119277056 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7213352876032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28412673 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184001447427584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179592763 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74733170 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3949 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142807217283072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179618973 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142807234322432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176707213742592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ÃUnIP2û- _wandbruntime‚4286  _runtime‚4286™)ÕQüË /loss/FRt3/assignment_loss‚5.457276773452759 0loss/FBt2/assignment_loss‚0.2339667081829475 0loss/SRqqt4/detection_loss‚1.220696068711971 epoch‚3 0loss/FRt2/assignment_loss‚3.6570123304020274 0loss/SRqqt1/assignment_loss‚2.36832549393177 1loss/SRqqt4/assignment_loss‚2.138487786316035 0loss/FBt3/assignment_loss‚0.4626052452792635 1loss/FBt4/assignment_loss‚0.27439678676637413 1loss/SRqqt2/detection_loss‚1.5224784009158612 /loss/FBt4/detection_loss‚0.7794111827604993 1loss/SRqqt3/assignment_loss‚2.379493706366595 0loss/FBt1/assignment_loss‚0.5447537008985819 1loss/SRqqt1/detection_loss‚2.5927790093421934 %loss/total_loss‚56.71324105126974 /loss/FRt1/assignment_loss‚5.110297307372093 /loss/FRt4/detection_loss‚1.9132109466367042 .loss/FBt1/detection_loss‚2.519359242171049 .loss/FBt3/detection_loss‚3.494867578148842 trainer/global_step‚3949 _step‚324 1loss/SRqqt2/assignment_loss‚2.249629584699869 /loss/FRt1/detection_loss‚2.6070349011570215 .loss/FBt2/detection_loss‚4.016617327928543 .loss/FRt3/detection_loss‚2.929009673992793 /loss/FRt2/detection_loss‚2.1653643900697883 /loss/SRqqt3/detection_loss‚2.81207975832855 _runtime‚4286.364074893 /loss/FRt4/assignment_loss‚3.264087147437609 % _timestamp‚1.7883992228043911e+094]¨¦ÞýÐ .loss/FBt3/detection_loss‚3.494867578148842 trainer/global_step‚3949 _step‚324 1loss/SRqqt2/assignment_loss‚2.249629584699869 /loss/FRt1/detection_loss‚2.6070349011570215 .loss/FBt2/detection_loss‚4.016617327928543 .loss/FRt3/detection_loss‚2.929009673992793 /loss/FRt2/detection_loss‚2.1653643900697883 /loss/SRqqt3/detection_loss‚2.81207975832855 _runtime‚4286.364074893 /loss/FRt4/assignment_loss‚3.264087147437609 % _timestamp‚1.7883992228043911e+09 /loss/FRt3/assignment_loss‚5.457276773452759 0loss/FBt2/assignment_loss‚0.2339667081829475 0loss/SRqqt4/detection_loss‚1.220696068711971 epoch‚3 0loss/FRt2/assignment_loss‚3.6570123304020274 0loss/SRqqt1/assignment_loss‚2.36832549393177 1loss/SRqqt4/assignment_loss‚2.138487786316035 0loss/FBt3/assignment_loss‚0.4626052452792635 1loss/FBt4/assignment_loss‚0.27439678676637413 1loss/SRqqt2/detection_loss‚1.5224784009158612 /loss/FBt4/d•ýExetection_loss‚0.7794111827604993 1loss/SRqqt3/assignment_loss‚2.379493706366595 0loss/FBt1/assignment_loss‚0.5447537008985819 1loss/SRqqt1/detection_loss‚2.5927790093421934 %loss/total_loss‚56.71324105126974 /loss/FRt1/assignment_loss‚5.110297307372093 /loss/FRt4/detection_loss‚1.9132109466367042 .loss/FBt1/detection_loss‚2.519359242171049Äöb{‘þ‚(:† úœãÔñÿÒ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3348 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,9£Kÿ‚(:À úœãÔÔÖ disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0& disk.nvme0n1.out‚51460.47705078125$ memory_percent‚98.91815185546875 disk.dm-1.out‚ 3090.31640625# disk./.usageGB‚1053.98193359375 disk.rbd2.in‚0 disk.drbd1777.in‚0 network.sent‚ 9.323382e+06 disk.sda3.out‚ 3090.31640625 disk.nvme0n1.in‚ 670.16796875# proc.memory.rssMB‚ 2846.33984375 cpu‚25.27215465327301 network.recv‚ 1.523542e+06 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 670.16796875# disk.dm-0.out‚51460.47705078125) disk./.usagePercent‚70.73361191322253 disk.sda1.out‚0! disk.rbd2.out‚130.89208984375 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 670.16796875( disk.nvme0c0n1.out‚51460.47705078125" proc.memory.availableMB‚177.25 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 2.20947265625 disk.sda.in‚ 0.07421875 disk.sda.out‚ 3090.31640625óÿb‘€‚(:† ‰ãÔòɆÓ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.324' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4648 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÛóQ‚(:Æ ‰ãÔœ¿°Õ disk.dm-1.out‚ 3090.66015625 disk.drbd1777.in‚0 disk.sda2.out‚0( disk./.usagePercent‚70.7336190814335 cpu‚25.271394130201354 disk.sda3.out‚ 3090.66015625 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚51460.86767578125 disk.rbd0.out‚0 disk.dm-0.in‚ 670.16796875 disk.sda.in‚ 0.07421875# memory_percent‚98.9168643951416* proc.memory.percent‚17.372679710388184 network.sent‚ 9.431282e+06% disk./.usageGB‚1053.9820404052734 disk.rbd1.out‚ 2.20947265625" disk.nvme0c0n1.in‚ 670.16796875& disk.nvme0n1.out‚51460.86767578125 disk.sda.out‚ 3090.66015625 disk.drbd1777.out‚0' proc.memory.availableMB‚ 177.4609375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚132.46630859375 disk.sda2.in‚0 network.recv‚ 1.529115e+06 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 670.16796875# disk.dm-0.out‚51460.86767578125 disk.rbd3.out‚0# proc.memory.rssMB‚ 2846.33984375„¯–ç‚‚(:„ ˜ãÔ¸ŠÊÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.449 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©žn2Rƒ‚(:Ç ˜ãÔóƒ¨Õ" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 670.16796875 cpu‚25.270573436942517 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3091.41015625( disk.nvme0c0n1.out‚51461.53564453125 disk.dm-1.in‚ 0.07421875 network.sent‚ 9.437624e+06 network.recv‚ 1.533499e+06 disk.rbd2.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚51461.53564453125# disk.dm-0.out‚51461.53564453125 disk.sda.out‚ 3091.41015625 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 177.4609375* proc.memory.percent‚17.372679710388184% disk./.usageGB‚1053.9819641113281 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0! disk.rbd2.out‚132.54443359375# proc.memory.rssMB‚ 2846.33984375) disk./.usagePercent‚70.73361396128281 disk.dm-0.in‚ 670.16796875 disk.sda2.in‚0 disk.rbd1.out‚ 2.20947265625# memory_percent‚98.9168643951416 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 3091.41015625̹22„- _wandbruntime‚4332  _runtime‚4332ÎÆ}÷Ç…Á _runtime‚4332.436446937 _step‚325 & lr-AdamW/pg1‚0.0001887506523545337 & lr-AdamW/pg2‚0.0001887506523545337 trainer/global_step‚3999 % _timestamp‚1.7883992688766546e+09Ö\.ÌÌ†Æ % _timestamp‚1.7883992688766546e+09 _runtime‚4332.436446937 _step‚325 & lr-AdamW/pg1‚0.0001887506523545337 & lr-AdamW/pg2‚0.0001887506523545337 trainer/global_step‚3999Ÿê x2‡- _wandbruntime‚4332  _runtime‚4332ΘËFà'ˆÚO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186216563854336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178832780317696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144518642297856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75623943 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28752581 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151798376186406 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160953442876416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186216274377728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178832804257792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7383494060032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153651678916096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73109513 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106166802 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144518625258496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160953694465024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7301781963264 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7279979897762 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75623941 _step‚326 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73110267 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7279750927910 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289476608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153651660913152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160953694465024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44357686 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106140620 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 % _timestamp‚1.7883992688773632e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181790745 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153651678916096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4332.437336002 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106166802 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265536512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153651660913152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7302015548928ÊÞ´å'‰ßO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44357684 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181764561 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7301781963264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75623943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7383759596544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 trainer/global_step‚3999 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7302015548928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160953442876416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106140620 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7279979897762 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75623941 _step‚326 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73110267 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7279750927910 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289476608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153651660913152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160953694465024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44357686 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106140620 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 % _timestamp‚1.7883992688773632e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181790745 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153651678916096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4332.437336002 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106166802 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265536512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153651660913152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7302015548928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186216563854336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178832780317696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144518642297856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75623943 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28752581 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151798376186406 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160953442876416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186216274377728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178832804257792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7383494060032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153651678916096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73109513 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106166802 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144518625258496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160953694465024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7301781963264 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28758713 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151835106613222 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7303591111680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75642120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287366144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178876159227392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153688821206016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44368337 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73127775 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186261765867520 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106192903 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153688803203072 _step‚327 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178876183167488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144553551951872 _runtime‚4333.327608518 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106192903 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7281554661350 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106166695 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160992645916672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 REDeviceStatsMonitor.on_train_ba’x˜X tch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚723 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7303824710656 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75642120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144553568991232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181808815 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181835025 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7281783631410 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106166695 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181835025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73127050 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263426048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75642122 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808Ve4ï&ŒéM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106166695 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160992645916672 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚723 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7303824710656 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75642120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144553568991232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181808815 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181835025 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7281783631410 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106166695 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181835025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73127050 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263426048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75642122 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 % _timestamp‚1.7883992697672093e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28758713 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151835106613222 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7303591111680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75642120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287366144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178876159227392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153688821206016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44368337 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73127775 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186261765867520 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106192903 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153688803203072 _step‚327 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178876183167488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144553551951872 _runtime‚4333.327608518 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106192903 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7281554661350 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696ÇÔf2- _wandbruntime‚4333  _runtime‚4333e0ˆvZŽÔ 0loss/FRt2/assignment_loss‚3.8993792816212305 1loss/FBt4/assignment_loss‚0.17674620979142877 .loss/FBt2/detection_loss‚2.500009232097202 /loss/FBt3/detection_loss‚2.6751569509506226 /loss/FRt3/assignment_loss‚5.440802507622298 1loss/SRqqt1/assignment_loss‚2.592581322659617 0loss/FBt2/assignment_loss‚0.3391087322023749 .loss/FRt1/detection_loss‚2.451052140621912 .loss/FBt1/detection_loss‚3.008964960391705 /loss/FBt4/detection_loss‚0.7051489683596984 &loss/total_loss‚54.080627868630124 2loss/SRqqt4/assignment_loss‚1.6938327063579817 0loss/SRqqt1/detection_loss‚3.094375105007835 trainer/global_step‚3999 /loss/FRt1/assignment_loss‚5.154641934803554 epoch‚3 2loss/SRqqt2/assignment_loss‚2.3967363951326925 1loss/SRqqt3/detection_loss‚2.4854529568127224 % _timestamp‚1.7883992697689936e+09 _step‚328 0loss/FRt4/assignment_loss‚2.4861171948058263 0loss/FBt1/assignment_loss‚0.5197210167859269 /loss/FRt3/detection_loss‚2.6182521138080332 0loss/FBt3/assignment_loss‚0.5015465803444386 /loss/FRt4/detection_loss‚1.8334888702790653 1loss/SRqqt2/detection_loss‚1.5940317371978034 _runtime‚4333.328669417 2loss/SRqqt3/assignment_loss‚2.4114488073757716 /loss/FRt2/detection_loss‚2.5119357273766867 1loss/SRqqt4/detection_loss‚0.9900964162236935†z(ö_Ù /loss/FRt4/detection_loss‚1.8334888702790653 1loss/SRqqt2/detection_loss‚1.5940317371978034 _runtime‚4333.328669417 2loss/SRqqt3/assignment_loss‚2.4114488073757716 /loss/FRt2/detection_loss‚2.5119357273766867 1loss/SRqqt4/detection_loss‚0.9900964162236935 0loss/FRt2/assignment_loss‚3.8993792816212305 1loss/FBt4/assignment_loss‚0.17674620979142877 .loss/FBt2/detection_loss‚2.500009232097202 /loss/FBt3/detection_loss‚2.6751569509506226 /loss/FRt3/assignment_loss‚5.440802507622298 1loss/SRqqt1/assignment_loss‚2.592581322659617 0loss/FBt2/assignment_loss‚0.3391087322023749 .loss/FRt1/detection_loss‚2.451052140621912 .loss/FBt1/detection_loss‚3.008964960391705 /loss/FBt4/detection_loss‚0.7051489683596984 &loss/total_loss‚54.080627868630124 2loss/SRqqt4/assignment_loss‚1.6938327063579817 0loss/SRqqt1/detection_loss‚3.094375105007835 trainer/global_step‚3999 /loss/FRt1/assignment_loss‚5.154641934803554 epoch‚3 2loss/SRqqt2/assignment_loss‚2.3967363951326925 1loss/SRqqt3/detection_loss‚2.4854529568127224 % _timestamp‚1.7883992697689936e+09 _step‚328 0loss/FRt4/assignment_loss‚2.4861171948058263 0loss/FBt1/assignment_loss‚0.5197210167859269 /loss/FRt3/detection_loss‚2.6182521138080332 0loss/FBt3/assignment_loss‚0.5015465803444386È^ß¿¯‘‚(:† §ãÔƒ†ßÓ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8412 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÕS‘‚(:È §ãÔŽáßÕ disk.drbd1777.out‚0 network.sent‚ 9.44299e+06 disk.sda.in‚ 0.07421875! disk.rbd2.out‚132.54443359375# proc.memory.rssMB‚ 2846.33984375* proc.memory.percent‚17.372679710388184 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚51560.60986328125 disk.drbd1777.in‚0 disk.dm-1.in‚ 0.07421875 cpu‚25.269866833061933 proc.cpu.threads‚17% disk./.usageGB‚1053.9812088012695 disk.dm-0.in‚ 670.16796875 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 177.47265625 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 2.20947265625 disk.dm-1.out‚ 3091.78515625 disk.nvme0n1.in‚ 670.16796875& disk.nvme0n1.out‚51560.60986328125 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 network.recv‚ 1.537259e+06) disk./.usagePercent‚70.73356327179103( disk.nvme0c0n1.out‚51560.60986328125 disk.rbd2.in‚0$ memory_percent‚98.91679286956787 disk.sda3.out‚ 3091.78515625" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd0.in‚0 disk.sda.out‚ 3091.78515625 !Ï’‚(:… ¶ãÔËÌæÒ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5296 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾¢DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73987642 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107473904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚162901029207040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76532891 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 287379456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162901029207040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153635157127626 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162901280795648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107473904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚155508793412096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚146264959927296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162901280795648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7369951191118 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7475193686016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7370180160970 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181001725802496N3Püæ›ßO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181001749742592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7474930246656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7392253797888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188476943428608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146264976966656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ {JÿWÿ!6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7392487383552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚4049 _runtime‚4379.465267081 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7392487383552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107447722 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 _step‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73987642 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107473904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚162901029207040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76532891 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 287379456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162901029207040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153635157127626 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162901280795648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107473904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚155508793412096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚146264959927296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162901280795648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7369951191118 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7475193686016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7370180160970 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181001725802496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 263439360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44890336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107447722 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29096609 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155508775409152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184006797Ê$¹c|œœ‚(:‘ ÔãÔ²·ÔÓ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.678' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.735599999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ëB‚(:· ÔãÔÏ™ÎÕ disk.dm-0.in‚ 670.16796875 disk.dm-0.out‚ 51660.6015625 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 cpu‚25.267566474311547 disk.rbd0.in‚0 disk.nvme0n1.in‚ 670.16796875( proc.memory.availableMB‚ 176.91796875* proc.memory.percent‚17.372679710388184 network.sent‚ 9.563571e+06 disk.rbd1.out‚ 2.20947265625 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0" disk.nvme0n1.out‚ 51660.6015625 disk.sda1.out‚0 network.recv‚ 1.552409e+06$ disk.nvme0c0n1.out‚ 51660.6015625 disk.sda.out‚ 3093.8203125 disk.rbd2.in‚0 disk.rbd2.out‚ 132.787109375# proc.memory.rssMB‚ 2846.33984375 proc.cpu.threads‚17$ disk./.usageGB‚1053.982334136963 disk.sda.in‚ 0.07421875 disk.rbd3.out‚0) disk./.usagePercent‚70.73363879401363 disk.sda3.out‚ 3093.8203125 disk.sda2.in‚0$ memory_percent‚98.92017841339111 disk.sda3.in‚ 0.07421875 disk.drbd1777.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.sda2.out‚0 disk.dm-1.out‚ 3093.820312561øW2ž- _wandbruntime‚4380  _runtime‚4380+ÞµÝê&ŸäM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107500005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162940232247296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7394296545280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107473797 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181045104712192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7476752699904 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184051077 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883993167936738e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚146299903660032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184024867 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107500005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44900987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155545935702016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188522153166848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155545917699072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295754752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 _runtime‚4380.354317455 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74004555 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76551072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162939980645376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153671887554650 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188521857412096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184051077 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76551072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153671641545230 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7394062946304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚181045128652288 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7394062946304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7477024514560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0a.×Ìï& éM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7371754924558 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181045104712192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7476752699904 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184051077 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883993167936738e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚146299903660032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184024867 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107500005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44900987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155545935702016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188522153166848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155545917699072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 295754752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 _runtime‚4380.354317455 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74004555 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76551072 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162939980645376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153671887554650 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188521857412096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184051077 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76551072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153671641545230 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7394062946304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚181045128652288 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7394062946304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7477024514560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107500005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162940232247296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7394296545280 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107473797 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚754 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108781006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155471446050422 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108754824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45422984 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7482725632512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164848615537664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4426.471730286 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7482959218176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77441841 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291573760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77441843 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164848615537664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _step‚334 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164848867126272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29443880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267633664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155471692059634 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77441841 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77441843 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7460151454326 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157365889905152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74866866 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29443126 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108781006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45422986 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7566580927488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2Í.yå'¯ßO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157365889905152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186222849 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7482959218176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157365907908096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7566313293824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186196665 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190736984581120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7460380424178 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 trainer/global_step‚4099 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7482725632512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77441841 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77441843 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7460151454326 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157365889905152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74866866 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29443126 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108781006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45422986 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7566580927488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148011294596096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 % _timestamp‚1.7883993629117193e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚754 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108781006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155471446050422 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108754824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45422984 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7482725632512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164848615537664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4426.471730286 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7482959218176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77441841 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291573760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77441843 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164848615537664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _step‚334 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164848867126272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29443880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267633664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155471692059634 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74883488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190782188813824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183214074137088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7462184157826 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7484534780928 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186267129 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108807107 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157403050198016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186240919 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 _runtime‚4427.369833249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186267129 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157403032195072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7484534780928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77460022 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7568138616832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186240919 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157403050198016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7484768379904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7568406237184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155508176477238 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77460020 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183214050196992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77460020 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190782480374272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788399363809511e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45433637 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155508422486658 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45433639 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7484768379904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148046238328832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29449851 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ŽZ;0î&²èM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190782480374272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.788399363809511e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45433637 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155508422486658 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45433639 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7484768379904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148046238328832 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29449851 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164887566976000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108807107 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148046221289472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7461955187766 _step‚335 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77460022 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74883488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190782188813824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183214074137088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7462184157826 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7484534780928 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186267129 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108807107 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157403050198016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186240919 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 _runtime‚4427.369833249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186267129 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157403032195072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7484534780928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77460022 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7568138616832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186240919 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157403050198016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7484768379904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7568406237184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155508176477238 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77460020 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚183214050196992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77460020 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7883994099556508e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157307980982430 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159223022404096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188438901 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110061926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29787380 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192997650308096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185339640712192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188412717 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159223004401152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7550580687386 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166796453456896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 _runtime‚4473.515647851 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166796201868288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7573197467136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192997360831488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149757646304256 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7658009595904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7573197467136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110088108 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110088108 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚724 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 trainer/global_step‚4149 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7573431052800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75743740 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265536512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159223004401152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166796453456896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75743014 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 _step‚338 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7657744059392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159223022404096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚726 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocatöÅqp—!ions.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7883994099556508e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157307980982430 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159223022404096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188438901 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110061926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29787380 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192997650308096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185339640712192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188412717 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159223004401152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7550580687386 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166796453456896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 _runtime‚4473.515647851 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166796201868288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7573197467136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192997360831488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149757646304256 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7658009595904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7573197467136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110088108 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110088108 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚724 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 trainer/global_step‚4149 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7573431052800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75743740 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265536512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159223004401152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166796453456896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75743014 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 _step‚338 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7657744059392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159223022404096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚726 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78368972 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166835404908544 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110088001 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45966287 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7575240214528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149792572997632 trainer/global_step‚4149 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75761472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166835153306624 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7575006615552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7575240214528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149792555958272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159260146691072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78368972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185382995681792 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188483181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7659572230656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188456971 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _runtime‚4474.416851954 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110114209 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193042567912448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _step‚339 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75760759 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7552155450974 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7575006615552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552384421034 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185383019621888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29794472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788399410856208e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159260164694016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110114209 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78368970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159260146691072 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29794472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788399410856208e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159260164694016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110114209 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78368970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159260146691072 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78368972 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166835404908544 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110088001 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45966287 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7575240214528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149792572997632 trainer/global_step‚4149 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75761472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166835153306624 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7575006615552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7575240214528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149792555958272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159260146691072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78368972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185382995681792 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188483181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7659572230656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188456971 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _runtime‚4474.416851954 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110114209 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193042567912448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _step‚339 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75760759 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7552155450974 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7575006615552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552384421034 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185383019621888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253ÓrÎÉ2Å- _wandbruntime‚4474  _runtime‚4474ñó¬™YÆÓ .loss/FRt1/assignment_loss‚5.14068671428796 2loss/SRqqt2/assignment_loss‚2.2972560583705635 0loss/FBt3/assignment_loss‚0.5865473367273808 .loss/FRt2/assignment_loss‚4.55200055700082 0loss/FBt1/assignment_loss‚0.5410375652599296 /loss/FBt3/detection_loss‚2.6246002197265623 _runtime‚4474.417945344 _step‚340 0loss/FRt3/assignment_loss‚5.1193250768324905 /loss/FRt1/detection_loss‚2.6194997231165567 .loss/FRt2/detection_loss‚2.322207697882102 1loss/SRqqt1/detection_loss‚2.4662746023386717 1loss/SRqqt3/detection_loss‚2.4026895036568514 /loss/FBt4/detection_loss‚0.6372165644703351 0loss/FBt2/assignment_loss‚0.7112976914271713 1loss/SRqqt2/detection_loss‚1.4846379018165696 .loss/FRt4/detection_loss‚2.103924921629104 1loss/SRqqt4/detection_loss‚1.1638228925673857 /loss/FBt1/detection_loss‚3.1582799933173438 /loss/FBt2/detection_loss‚3.3498493432998657 epoch‚3 trainer/global_step‚4149 0loss/FRt4/assignment_loss‚2.6156263202428818 %loss/total_loss‚56.16790783093299 1loss/SRqqt1/assignment_loss‚3.023262906819582 2loss/SRqqt3/assignment_loss‚2.3819393853883484 2loss/SRqqt4/assignment_loss‚1.7975331547035687 1loss/FBt4/assignment_loss‚0.25644777807822366 /loss/FRt3/detection_loss‚2.8119439219727234 % _timestamp‚1.7883994108582218e+09zª •^ÇØ .loss/FRt1/assignment_loss‚5.14068671428796 2loss/SRqqt2/assignment_loss‚2.2972560583705635 0loss/FBt3/assignment_loss‚0.5865473367273808 .loss/FRt2/assignment_loss‚4.55200055700082 0loss/FBt1/assignment_loss‚0.5410375652599296 /loss/FBt3/detection_loss‚2.6246002197265623 _runtime‚4474.417945344 _step‚340 0loss/FRt3/assignment_loss‚5.1193250768324905 /loss/FRt1/detection_loss‚2.6194997231165567 .loss/FRt2/detection_loss‚2.322207697882102 1loss/SRqqt1/detection_loss‚2.4662746023386717 1loss/SRqqt3/detection_loss‚2.4026895036568514 /loss/FBt4/detection_loss‚0.6372165644703351 0loss/FBt2/assignment_loss‚0.7112976914271713 1loss/SRqqt2/detection_loss‚1.4846379018165696 .loss/FRt4/detection_loss‚2.103924921629104 1loss/SRqqt4/detection_loss‚1.1638228925673857 /loss/FBt1/detection_loss‚3.1582799933173438 /loss/FBt2/detection_loss‚3.3498493432998657 epoch‚3 trainer/global_step‚4149 0loss/FRt4/assignment_loss‚2.6156263202428818 %loss/total_loss‚56.16790783093299 1loss/SRqqt1/assignment_loss‚3.023262906819582 2loss/SRqqt3/assignment_loss‚2.3819393853883484 2loss/SRqqt4/assignment_loss‚1.7975331547035687 1loss/FBt4/assignment_loss‚0.25644777807822366 /loss/FRt3/detection_loss‚2.8119439219727234 % _timestamp‚1.7883994108582218e+09Ô™ÞzŽÈ‚(:ƒ ½žãÔ“ÜçÒ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚87.81' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.562 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—8£ªÉ‚(:¿ ½žãÔŠîÌÕ* proc.memory.percent‚17.372679710388184) disk./.usagePercent‚70.73363290584035 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.sda.out‚ 3097.91796875 proc.cpu.threads‚17 network.sent‚ 9.889653e+06% disk./.usageGB‚1053.9822463989258 disk.sda1.in‚0! disk.rbd2.out‚136.29150390625 disk.sda3.out‚ 3097.91796875 disk.dm-1.in‚ 0.07421875# proc.memory.rssMB‚ 2846.33984375 network.recv‚ 1.58653e+06 disk.sda1.out‚0 disk.dm-0.in‚ 670.16796875 disk.sda.in‚ 0.07421875( proc.memory.availableMB‚ 176.44921875 disk.sda2.out‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚51961.51171875# disk.nvme0n1.out‚51961.51171875 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 3097.91796875 disk.dm-0.out‚51961.51171875 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd0Û¯¤æ .in‚0 disk.nvme0n1.in‚ 670.16796875$ memory_percent‚98.92303943634033 cpu‚25.262503116321962 disk.sda2.in‚0 disk.rbd1.out‚ 2.20947265625h]$ÕÊ‚(:… ÌžãÔ°°ÂÓ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.085' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.417 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À0†HË‚(:½ ÌžãÔåÌÕ$ disk.nvme0c0n1.out‚ 51961.8671875 disk.dm-1.out‚ 3098.48046875 network.recv‚ 1.593246e+06) disk./.usagePercent‚70.73244631091914 disk.nvme0n1.in‚ 670.16796875 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 670.16796875" disk.nvme0n1.out‚ 51961.8671875 disk.sda1.in‚0# proc.memory.rssMB‚ 2846.33984375* proc.memory.percent‚17.372679710388184 cpu‚25.261798911281872 disk.rbd2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 176.32421875 disk.sda.out‚ 3098.48046875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0$ memory_percent‚98.92380237579346 disk.dm-0.in‚ 670.16796875 disk.sda.in‚ 0.07421875! disk.rbd2.out‚136.43212890625 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 51961.8671875 disk.rbd3.in‚0 disk.sda3.out‚ 3098.48046875 disk.rbd1.out‚ 2.20947265625% disk./.usageGB‚1053.9645652770996 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 network.sent‚ 9.998632e+06 disk.rbd0.in‚0 disk.rbd0.out‚0+TÛ‘Ì‚(:† ÛžãÔ€Û…Ó gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.558' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7116 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BizÅLÍ‚(:Á ÛžãÔ—ìÿÕ network.recv‚ 1.596422e+06 disk.nvme0n1.in‚ 670.16796875 disk.dm-0.in‚ 670.16796875! disk.dm-0.out‚52061.220703125 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875% disk./.usageGB‚1053.9646110534668 disk.dm-1.out‚ 3098.9296875 disk.sda.in‚ 0.07421875 disk.sda2.out‚0$ disk.nvme0n1.out‚52061.220703125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 2.20947265625* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17 network.sent‚ 1.0003781e+07) disk./.usagePercent‚70.73244938300955 disk.dm-1.in‚ 0.07421875 disk.drbd1777.in‚0 disk.sda3.out‚ 3098.9296875# proc.memory.rssMB‚ 2846.33984375 disk.sda1.in‚0 disk.rbd1.in‚0$ memory_percent‚98.92394542694092" disk.nvme0c0n1.in‚ 670.16796875& disk.nvme0c0n1.out‚52061.220703125 disk.sda.out‚ 3098.9296875 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚136.43212890625 disk.sda2.in‚0( proc.memory.availableMB‚ 176.30078125 cpu‚25.261094912404054 disk.rbd0.in‚0ÏPŽ2Î- _wandbruntime‚4520  _runtime‚4520Q1 %ÈÏ $ _timestamp‚1.788399456828823e+09 _runtime‚4520.388645921 _step‚341 ' lr-AdamW/pg1‚0.00018398071902775458 ' lr-AdamW/pg2‚0.00018398071902775458 trainer/global_step‚4199ï­ÍÐÇ $ _timestamp‚1.788399456828823e+09 _runtime‚4520.388645921 _step‚341 ' lr-AdamW/pg1‚0.00018398071902775458 ' lr-AdamW/pg2‚0.00018398071902775458 trainer/global_step‚4199ÕW ¼®2Ñ- _wandbruntime‚4520  _runtime‚4520ËCŠà'ÒÚO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159144515914438 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7663669301760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161080118897152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195257701433344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187508562256896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151503963933696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190628769 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168743788198912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30132664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79259743 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267633664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151503980973056 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79259743 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161080136900096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7640780950594 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7749139176448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168743788198912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111395210 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168744039787520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190654953 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195257993007104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111395210 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7663902887424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161080136900096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79259741ús å'ÓßO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7640780950594 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7749139176448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168743788198912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111395210 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168744039787520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190654953 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195257993007104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111395210 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7663902887424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161080136900096 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79259741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159144515914438 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7663669301760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161080118897152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195257701433344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187508562256896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151503963933696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190628769 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168743788198912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30132664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79259743 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267633664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151503980973056 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79259743 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161080136900096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32Ö²wc2Ô- _wandbruntime‚4521  _runtime‚4521ÔB¦`!ÕäM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168782739637248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79277922 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190699233 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195302925441024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161117261187072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79277920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7750984274432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111421311 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111421311 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161117261187072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7665712049152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30140270 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159181246341254 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187551965106688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7751247700480 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168782991239168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 _runtime‚4521.276827675 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7665712049152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161117279190016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111395103 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287366144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7642355714182 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46498939 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111395103 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190673023 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168782991239168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7665478450176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190673023 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76639209 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30139577 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 _step‚343 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7665478450176 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76638514 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263426048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187551941166592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151538907666432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 trainer/global_step‚4199¢¥!Úï&ÖéM _step‚343 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7665478450176 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76638514 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263426048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187551941166592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151538907666432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 trainer/global_step‚4199 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168782739637248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79277922 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190699233 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195302925441024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161117261187072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79277920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7750984274432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111421311 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111421311 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161117261187072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7665712049152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30140270 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159181246341254 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187551965106688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7751247700480 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168782991239168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 _runtime‚4521.276827675 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7665712049152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161117279190016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111395103 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287366144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7642355714182 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46498939 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111395103 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190673023 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168782991239168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7665478450176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190673023 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76639209 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30139577 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307×oÅî)2×- _wandbruntime‚4521  _runtime‚4521ÝùË©VØÐ 2loss/SRqqt2/assignment_loss‚2.3275855320848917 1loss/SRqqt1/detection_loss‚2.2885021321913777 _step‚344 2loss/SRqqt3/assignment_loss‚2.4361284549037614 .loss/FBt1/detection_loss‚2.577460891008377 2loss/SRqqt4/assignment_loss‚1.9099773642917475 0loss/FBt1/assignment_loss‚0.5045016810756351 -loss/FRt1/detection_loss‚2.36571197360754 /loss/FRt2/detection_loss‚2.3443704735725484 /loss/FRt3/detection_loss‚2.5809017032384873 trainer/global_step‚4199 /loss/FRt1/assignment_loss‚5.251467400789261 /loss/FRt4/assignment_loss‚3.268791928291321 0loss/FBt2/assignment_loss‚0.5713175671796004 1loss/FBt4/assignment_loss‚0.23399605540303997 /loss/FBt3/detection_loss‚3.6683178742726645 _runtime‚4521.277983182 1loss/SRqqt2/detection_loss‚1.5727852596656273 1loss/SRqqt3/detection_loss‚2.4210443016555576 1loss/SRqqt4/detection_loss‚1.2785522416234016 epoch‚3 /loss/FRt3/assignment_loss‚5.256617850065231 /loss/FBt3/assignment_loss‚0.795473669966062 .loss/FBt2/detection_loss‚2.770978113015493 %loss/total_loss‚56.11936591895987 % _timestamp‚1.7883994577182555e+09 0loss/FRt2/assignment_loss‚3.9691486967370864 /loss/FRt4/detection_loss‚1.9470643819868565 .loss/FBt4/detection_loss‚0.757177696510982 1loss/SRqqt1/assignment_loss‚3.0214926758233243“æ·[ÙÕ 1loss/SRqqt3/detection_loss‚2.4210443016555576 1loss/SRqqt4/detection_loss‚1.2785522416234016 epoch‚3 /loss/FRt3/assignment_loss‚5.256617850065231 /loss/FBt3/assignment_loss‚0.795473669966062 .loss/FBt2/detection_loss‚2.770978113015493 %loss/total_loss‚56.11936591895987 % _timestamp‚1.7883994577182555e+09 0loss/FRt2/assignment_loss‚3.9691486967370864 /loss/FRt4/detection_loss‚1.9470643819868565 .loss/FBt4/detection_loss‚0.757177696510982 1loss/SRqqt1/assignment_loss‚3.021492675823324 2loss/SRqqt2/assignment_loss‚2.3275855320848917 1loss/SRqqt1/detection_loss‚2.2885021321913777 _step‚344 2loss/SRqqt3/assignment_loss‚2.4361284549037614 .loss/FBt1/detection_loss‚2.577460891008377 2loss/SRqqt4/assignment_loss‚1.9099773642917475 0loss/FBt1/assignment_loss‚0.5045016810756351 -loss/FRt1/detection_loss‚2.36571197360754 /loss/FRt2/detection_loss‚2.3443704735725484 /loss/FRt3/detection_loss‚2.5809017032384873 trainer/global_step‚4199 /loss/FRt1/assignment_loss‚5.251467400789261 /loss/FRt4/assignment_loss‚3.268791928291321 0loss/FBt2/assignment_loss‚0.5713175671796004 1loss/FBt4/assignment_loss‚0.23399605540303997 /loss/FBt3/detection_loss‚3.6683178742726645 _runtime‚4521.277983182 1loss/SRqqt2/detection_loss‚1.5727852596656273Ø1r—lÚ‚(:… êžãÔÓûüÒ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.565' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.113 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Y;^AÛ‚(:¶ êžãÔÜ°ÅÖ disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 3099.53125 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 cpu‚25.26039367785101 network.sent‚ 1.0010227e+07) disk./.usagePercent‚70.73244810297187& disk.nvme0c0n1.out‚52061.591796875 disk.sda3.in‚ 0.07421875# proc.memory.rssMB‚ 2846.33984375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 670.16796875 disk.dm-0.in‚ 670.16796875 disk.sda.out‚ 3099.53125 disk.sda1.out‚0 disk.rbd2.out‚136.5966796875 disk.rbd0.in‚0$ disk.nvme0n1.out‚52061.591796875 disk.drbd1777.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 670.16796875! disk.dm-0.out‚52061.591796875 disk.rbd2.in‚0# memory_percent‚98.9262580871582 disk.rbd1.out‚ 2.20947265625& proc.memory.availableMB‚ 175.921875 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 3099.53125* proc.memory.percent‚17.372679710388184% disk./.usageGB‚1053.9645919799805 disk.drbd1777.in‚0 network.recv‚ 1.601227e+06 disk.rbd1.in‚0u±é œÜ‚(:‘ ùžãÔÖûÒ gpu.0.gpu‚41.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.205' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.441000000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K,æHMÝ‚(: ùžãÔüà©Õ% disk./.usageGB‚1053.9646759033203 disk.drbd1777.in‚0 disk.rbd3.in‚0$ memory_percent‚98.92621040344238 network.recv‚ 1.606059e+06) disk./.usagePercent‚70.73245373513764 disk.sda3.out‚ 3099.96484375" disk.nvme0c0n1.in‚ 670.16796875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0& disk.nvme0c0n1.out‚52062.248046875 disk.dm-1.out‚ 3099.96484375 disk.nvme0n1.in‚ 670.16796875 disk.sda.out‚ 3099.96484375 network.sent‚ 1.0117222e+07 disk.rbd0.in‚0 disk.dm-0.in‚ 670.16796875* proc.memory.percent‚17.372679710388184 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 disk.sda1.in‚0' proc.memory.availableMB‚ 175.9296875 disk.sda2.in‚0! disk.dm-0.out‚52062.248046875 disk.rbd2.out‚136.5966796875 cpu‚25.259662539618862 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 2.20947265625$ disk.nvme0n1.out‚52062.248046875 disk.drbd1777.out‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2846.33984375 disk.sda2.out‚0 disk.rbd1.in‚0µïäÞ‚(:… ˆŸãÔ´ÄæÒ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.285' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.057 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XÄ:Sß‚(:È ˆŸãÔò¥›Õ disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 670.16796875 disk.sda.out‚ 3100.62109375 proc.cpu.threads‚17 network.recv‚ 1.610324e+06 disk.rbd1.in‚0 disk.dm-1.out‚ 3100.62109375) disk./.usagePercent‚70.73245399114518# proc.memory.rssMB‚ 2846.33984375 network.sent‚ 1.0123521e+07% disk./.usageGB‚1053.9646797180176 disk.sda1.out‚0 disk.rbd2.out‚138.2490234375$ memory_percent‚98.92621040344238 disk.drbd1777.in‚0 disk.dm-0.in‚ 670.16796875' proc.memory.availableMB‚ 175.9296875* proc.memory.percent‚17.372679710388184 cpu‚25.259032695940306 disk.rbd3.out‚0# disk.dm-0.out‚52161.44970703125 disk.sda.in‚ 0.07421875 disk.rbd3.in‚0 disk.sda3.out‚ 3100.62109375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.16796875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 2.20947265625( disk.nvme0c0n1.out‚52161.44970703125 disk.dm-1.in‚ 0.07421875& disk.nvme0n1.out‚52161.44970703125 ŸàZ2à- _wandbruntime‚4567  _runtime‚4567HkH0Éáà _runtime‚4567.047297717 _step‚345 ' lr-AdamW/pg1‚0.00018274984777462042 ' lr-AdamW/pg2‚0.00018274984777462042 trainer/global_step‚4249 % _timestamp‚1.7883995034874756e+09Ï~,uÎâÈ _step‚345 ' lr-AdamW/pg1‚0.00018274984777462042 ' lr-AdamW/pg2‚0.00018274984777462042 trainer/global_step‚4249 % _timestamp‚1.7883995034874756e+09 _runtime‚4567.047297717Ùyèy2ã- _wandbruntime‚4567  _runtime‚4567½øíà'äÚO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77499079 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47020936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170691374529536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80168691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30478145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265536512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7730981213802 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170691626118144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7754374722048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112702312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 % _timestamp‚1.7883995034881895e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153250298602496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112702312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80168693 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170691374529536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7754374722048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112676130 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚745 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189677531681792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160981296855658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192871005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153250315641856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170691626118144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30478890 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189677507741696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289476608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 _step‚346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162937251396096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7730752243950 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80168691 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197518038455808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47020934 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80168693 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77499826 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7840796250624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162937251396096 _runtime‚4567.048260711 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7754141136384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197518327932416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112676130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7754141136384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160981050846446 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162937233393152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192844821 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0‘|ûäµ åßO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197518327932416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112676130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7754141136384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160981050846446 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162937233393152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192844821 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77499079 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47020936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170691374529536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80168691 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30478145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265536512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7730981213802 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 ZKDeviceStatsMonitor.on_traoº:20in_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170691626118144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7754374722048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112702312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 % _timestamp‚1.7883995034881895e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153250298602496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112702312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80168693 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170691374529536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7754374722048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112676130 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚745 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189677531681792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160981296855658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192871005 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153250315641856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170691626118144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30478890 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189677507741696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289476608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 _step‚346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162937251396096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7730752243950 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80168691 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197518038455808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47020934 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80168693 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77499826 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7840796250624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162937251396096 _runtime‚4567.048260711 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7754141136384ÚåqÒÏ2æ- _wandbruntime‚4567  _runtime‚4567eLi‹ê&çäM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192915285 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162974393686016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192889075 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7756183883776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170730577569792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263426048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153285225295872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80186872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192889075 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197563242948096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161018027282682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161017781273262 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112728413 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192915285 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170730577569792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 trainer/global_step‚4249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77516832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189720910591488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7732784947450 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153285242335232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162974375683072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287366144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162974393686016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112728413 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 % _timestamp‚1.7883995043725483e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197563530314240ƒÊ? ï&èéM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112702205 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77517549 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7842619722752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7755950284800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170730325967872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30485960 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153285242335232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162974375683072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 287366144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162974393686016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112728413 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 % _timestamp‚1.7883995043725483e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197563530314240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192915285 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162974393686016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192889075 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7756183883776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170730577569792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 263426048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153285225295872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80186872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192889075 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197563242948096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161018027282682 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161017781273262 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112728413 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192915285 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170730577569792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 trainer/global_step‚4249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77516832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189720910591488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7732784947450 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 _step‚350 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚199778432576512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78377830 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7931979350016 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172638960860160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172638960860160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 _runtime‚4614.001899888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30824246 trainer/global_step‚4299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7844612971008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81077643 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81077641 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78378562 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7932249080832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195060873 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191846453226496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162817585778454 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269730816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172639212448768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164794347889152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191846477166592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81077643 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195087057 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154996633271296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0^dØÄå'÷ßO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164794365892096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114009414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7844612971008 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162817585778454 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 269730816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172639212448768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164794347889152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191846477166592 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81077643 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195087057 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154996633271296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30824976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7821181477010 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81077641 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164794365892096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113983232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162817831787666 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293670912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7844846556672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚199778726247424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 _step‚350 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚199778432576512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78377830 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7931979350016 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172638960860160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172638960860160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 _runtime‚4614.001899888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30824246 trainer/global_step‚4299 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7844612971008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81077643 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81077641 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78378562 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7932249080832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195060873 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191846453226496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81095822 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7933808455168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7822756240598 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195105127 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7822985210658 % _timestamp‚1.7883995513367255e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172677912298496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30831131 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289463296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81095820 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚199823640591360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195131337 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81095820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164831490179072 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7846655718400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30831821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47564237 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78395368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81095822 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195131337 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7934073978368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164831490179072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78396060 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523200 trainer/global_step‚4299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172678163900416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114035515 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114009307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155031577004032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191889832136192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155031559964672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78395368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81095822 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195131337 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7934073978368 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164831490179072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78396060 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523200 trainer/global_step‚4299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172678163900416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114035515 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114009307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155031577004032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191889832136192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155031559964672 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81095822 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7933808455168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7822756240598 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195105127 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7822985210658 % _timestamp‚1.7883995513367255e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172677912298496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30831131 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 289463296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81095820 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚199823640591360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195131337 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81095820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164831490179072 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7846655718400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30831821 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚351 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47564237 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976ßìF2û- _wandbruntime‚4614  _runtime‚4614‡¿Å®RüÌ /loss/SRqqt3/detection_loss‚2.47567714798835 0loss/FRt2/assignment_loss‚3.7429430693387986 /loss/FRt2/detection_loss‚2.5678735189139843 .loss/FRt3/detection_loss‚2.706563760340214 1loss/SRqqt2/detection_loss‚1.7828470317285452 .loss/FBt4/detection_loss‚0.527536921384858 0loss/FRt4/assignment_loss‚2.5178596747888102 0loss/SRqqt1/detection_loss‚2.663739305521761 0loss/FBt1/assignment_loss‚0.7734495743542749 0loss/FBt2/assignment_loss‚0.7069267351180315 .loss/FRt4/detection_loss‚2.118749396623792 epoch‚3 % _timestamp‚1.7883995513386633e+09 2loss/SRqqt2/assignment_loss‚2.3478394924704706 1loss/SRqqt4/assignment_loss‚1.807895298699538 0loss/FBt3/assignment_loss‚0.6207569958642125 .loss/FBt3/detection_loss‚3.524363249540329 /loss/FRt1/assignment_loss‚4.210635011846369 .loss/FRt3/assignment_loss‚4.77215678691864 2loss/SRqqt3/assignment_loss‚2.7075233767109532 trainer/global_step‚4299 _step‚352 1loss/SRqqt1/assignment_loss‚2.870471153408289 %loss/total_loss‚54.58193641018755 _runtime‚4614.898356414 .loss/FRt1/detection_loss‚2.659424852241169 1loss/SRqqt4/detection_loss‚1.2305852099259695 /loss/FBt1/detection_loss‚2.4481896368356852 .loss/FBt2/detection_loss‚2.523631364107132 1loss/FBt4/assignment_loss‚0.27429784551737535ˆ~OWýÑ 1loss/SRqqt1/assignment_loss‚2.870471153408289 %loss/total_loss‚54.58193641018755 _runtime‚4614.898356414 .loss/FRt1/detection_loss‚2.659424852241169 1loss/SRqqt4/detection_loss‚1.2305852099259695 /loss/FBt1/detection_loss‚2.4481896368356852 .loss/FBt2/detection_loss‚2.523631364107132 1loss/FBt4/assignment_loss‚0.27429784551737535 /loss/SRqqt3/detection_loss‚2.47567714798835 0loss/FRt2/assignment_loss‚3.7429430693387986 /loss/FRt2/detection_loss‚2.5678735189139843 .loss/FRt3/detection_loss‚2.706563760340214 1loss/SRqqt2/detection_loss‚1.7828470317285452 .loss/FBt4/detection_loss‚0.527536921384858 0loss/FRt4/assignment_loss‚2.5178596747888102 0loss/SRqqt1/detection_loss‚2.663739305521761 0loss/FBt1/assignment_loss‚0.7734495743542749 0loss/FBt2/assignment_loss‚0.7069267351180315 .loss/FRt4/detection_loss‚2.118749396623792 epoch‚3 % _timestamp‚1.7883995513386633e+09 2loss/SRqqt2/assignment_loss‚2.3478394924704706 1loss/SRqqt4/assignment_loss‚1.807895298699538 0loss/FBt3/assignment_loss‚0.6207569958642125 .loss/FBt3/detection_loss‚3.524363249540329 /loss/FRt1/assignment_loss‚4.210635011846369 .loss/FRt3/assignment_loss‚4.77215678691864 2loss/SRqqt3/assignment_loss‚2.7075233767109532 trainer/global_step‚4299 _step‚352àšWWþ‚(:… ÄŸãÔÞ‡ïÔ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚81.418' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.2836 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÒ}Kÿ‚(:À ÄŸãÔ¿þîÕ disk.sda2.in‚0 disk.sda3.out‚ 3102.890625 disk.dm-1.out‚ 3102.890625 disk.rbd0.in‚0* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17 network.recv‚ 1.628943e+06 disk.dm-0.in‚ 670.16796875 disk.sda1.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.nvme0n1.in‚ 670.16796875# proc.memory.rssMB‚ 2846.33984375 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0$ memory_percent‚98.92292022705078 cpu‚25.25627225423146) disk./.usagePercent‚70.73245783125819 disk.sda3.in‚ 0.07421875 disk.sda1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚52262.56689453125 disk.rbd3.out‚0 disk.dm-1.in‚ 0.07421875! disk.rbd2.out‚138.49169921875 network.sent‚ 1.0249344e+07% disk./.usageGB‚1053.9647369384766# disk.dm-0.out‚52262.56689453125 disk.sda.out‚ 3102.890625 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 2.20947265625 disk.rbd0.out‚0& disk.nvme0n1.out‚52262.56689453125% proc.memory.availableMB‚ 176.46875 disk.rbd2.in‚0Éâ½Àœ€‚(:‘ ÓŸãÔø³Ó gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.723' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.144599999999997 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘i¢Q‚(:Æ ÓŸãÔÄ£ÅÕ network.sent‚ 1.0357201e+07 disk.drbd1777.in‚0 proc.cpu.threads‚17 network.recv‚ 1.634793e+06) disk./.usagePercent‚70.73245808726571 disk.sda.in‚ 0.07421875% disk./.usageGB‚1053.9647407531738( disk.nvme0c0n1.out‚52362.02783203125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2846.33984375 disk.dm-1.out‚ 3103.4140625& disk.nvme0n1.out‚52362.02783203125 disk.dm-0.in‚ 670.16796875 disk.sda.out‚ 3103.4140625 disk.sda1.out‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0 disk.rbd1.out‚ 2.20947265625 disk.rbd0.out‚0 disk.drbd1777.out‚0! disk.rbd2.out‚140.06591796875 disk.sda2.in‚0 disk.sda3.out‚ 3103.4140625( proc.memory.availableMB‚ 176.41796875* proc.memory.percent‚17.372679710388184" disk.nvme0c0n1.in‚ 670.16796875 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 670.16796875# disk.dm-0.out‚52362.02783203125$ memory_percent‚98.92323017120361 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 cpu‚25.25555861628782 disk.rbd3.out‚0ÁÙÄ‘‚‚(:… âŸãÔУÄÔ gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚84.468' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.8936 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J•œ7Iƒ‚(:¾ âŸãÔÏÒ¹Õ disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.16796875 disk.sda1.out‚0 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd3.in‚0 disk.sda3.out‚ 3103.9375 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0! disk.rbd2.out‚140.19091796875# proc.memory.rssMB‚ 2846.33984375( disk.nvme0c0n1.out‚52362.32861328125 disk.sda2.in‚0 disk.dm-0.in‚ 670.16796875 disk.rbd1.in‚0 disk.dm-1.out‚ 3103.9375$ memory_percent‚98.92475605010986& disk.nvme0n1.out‚52362.32861328125 network.sent‚ 1.0363506e+07 network.recv‚ 1.639288e+06 disk.rbd3.out‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 176.16796875 cpu‚25.254846839390222) disk./.usagePercent‚70.73245808726571 disk.sda2.out‚0# disk.dm-0.out‚52362.32861328125 proc.cpu.threads‚17 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 2.20947265625 disk.sda.in‚ 0.07421875 disk.sda.out‚ 3103.9375 disk.sda1.in‚0* proc.memory.percent‚17.372679710388184% disk./.usageGB‚1053.9647407531738 disk.drbd1777.in‚0û Þ–2„- _wandbruntime‚4660  _runtime‚4660£¦‹É…à trainer/global_step‚4349 % _timestamp‚1.7883995974344544e+09 _runtime‚4660.994274746 _step‚353 ' lr-AdamW/pg1‚0.00018024382418004915 ' lr-AdamW/pg2‚0.00018024382418004915í dÎ†È % _timestamp‚1.7883995974344544e+09 _runtime‚4660.994274746 _step‚353 ' lr-AdamW/pg1‚0.00018024382418004915 ' lr-AdamW/pg2‚0.00018024382418004915 trainer/global_step‚4349áWGÜ2‡- _wandbruntime‚4660  _runtime‚4660 /#Çà'ˆÚO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164654366719674 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115316516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197303109 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48086234 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7935318391296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31169807 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166651462385152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79256762 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166651480388096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48086236 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194015398711296 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7935084805632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202039117868032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7935084805632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7883995974351792e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156742984979456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174586547190784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174586547190784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81986593 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7911152770366 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174586798779392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194015422651392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164654120710462 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166651462385152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166651480388096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81986591 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271827968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516ìM–‰ßO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295768064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166651480388096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48086236 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194015398711296 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7935084805632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202039117868032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 WEDeviceStatsMonÅVOãitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7935084805632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7883995974351792e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156742984979456 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174586547190784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174586547190784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81986593 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7911152770366 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174586798779392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194015422651392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164654120710462 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166651462385152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166651480388096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81986591 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271827968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164654366719674 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115316516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197303109 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48086234 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7935318391296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31169807 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166651462385152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79256762 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164690851137278 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197321179 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174625750231040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48096887 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197347389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166688622678016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4661.886847673 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚725 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156777911672832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115342617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31176897 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202084026756096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156777894633472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚4349 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174625750231040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197347389 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82004770 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79274511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166688622678016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166688604675072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7937127553024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7937127553024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8025516755456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8025249135104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚355 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194058777620992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48096889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7912956503806 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82004772 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115316409 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174625498629120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82004772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚727 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194058801561088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79273784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 % _timestamp‚1.7883995983262303e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79274511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166688622678016 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166688604675072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7937127553024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7937127553024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8025516755456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8025249135104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚355 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194058777620992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48096889 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7912956503806 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82004772 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115316409 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174625498629120 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82004772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚727 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194058801561088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79273784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 % _timestamp‚1.7883995983262303e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164690851137278 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197321179 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174625750231040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48096887 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197347389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166688622678016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4661.886847673 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚725 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156777911672832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115342617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31176897 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202084026756096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156777894633472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚4349 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174625750231040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197347389 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82004770 F‹û¨È˜Â % lr-AdamW/pg2‚0.000178969194090173 trainer/global_step‚4399 # _timestamp‚1.78839964448649e+09 _runtime‚4708.046313041 _step‚357 % lr-AdamW/pg1‚0.000178969194090173å3ú U2™- _wandbruntime‚4708  _runtime‚4708ÑŒ ‡ß'šÙO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176534385110016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116623618 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116597436 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82895541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31515616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001582003426 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168508594884096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8025790225920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158489302608896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176534133521408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295768064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168508576881152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204299164957696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚358 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80133771 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48618884 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166490655642470 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168508594884096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 $ _timestamp‚1.788399644487267e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80134502 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8001353033574 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82895543 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204299460725760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚4708.047335508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8114820761600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82895541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199519161 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82895543 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 trainer/global_step‚4399 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8025790225920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8025556640256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116597436 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271827968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199492977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2îV8ä'›ÞO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80134502 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8001353033574 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82895543 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204299460725760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 _runtime‚4708.047335508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8114820761600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82895541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199519161 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82895543 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 trainer/global_step‚4399 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8025790225920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8025556640256 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116597436 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 271827968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199492977 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176534385110016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116623618 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116597436 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82895541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31515616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001582003426 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168508594884096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8025790225920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158489302608896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176534133521408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 295768064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168508576881152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204299164957696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚358 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80133771 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48618884 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166490655642470 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168508594884096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 $ _timestamp‚1.788399644487267e+09 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8027599387648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168545737174016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8027365788672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚750 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80151058 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168545719171072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚196227723105792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8003385737074 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166527632078706 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82913720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 _step‚359 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176573084959744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176573336561664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204344662067200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204344368409600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158524229302272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 trainer/global_step‚4399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176573336561664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8027599387648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82913722 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116623511 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31522271 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82913720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48629537 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8003156767014 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199563441 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80151810 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158524246341632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8116915021312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82913722 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116623511 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116649719 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166527386069286 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48629539 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 % _timestamp‚1.7883996453809097e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199537231 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116649719 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 _runtime‚4708.941536616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199537231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224NÜ]:,žéM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8027599387648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168545737174016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8027365788672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚750 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80151058 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168545719171072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚196227723105792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8003385737074 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166527632078706 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82913720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 _step‚359 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176573084959744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176573336561664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204344662067200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204344368409600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158524229302272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 trainer/global_step‚4399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176573336561664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8027599387648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82913722 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116623511 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31522271 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82913720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48629537 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8003156767014 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199563441 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80151810 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158524246341632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8116915021312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82913722 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116623511 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116649719 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166527386069286 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48629539 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 % _timestamp‚1.7883996453809097e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199537231 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116649719 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 _runtime‚4708.941536616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199537231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0çÛ’À2Ÿ- _wandbruntime‚4708  _runtime‚4708_f÷T Î 1loss/SRqqt3/detection_loss‚2.3012915501991906 trainer/global_step‚4399 /loss/FRt2/assignment_loss‚4.619996865590413 1loss/FBt4/assignment_loss‚0.27741776494768755 0loss/FBt1/assignment_loss‚0.3319794340215075 1loss/SRqqt1/detection_loss‚2.2962625332780786 1loss/SRqqt2/detection_loss‚1.8834795432706033 /loss/FBt1/detection_loss‚3.5785135626792908 /loss/FBt3/detection_loss‚2.7424696021609836 %loss/total_loss‚54.05738419771303 .loss/FRt2/detection_loss‚2.548271872294255 0loss/FRt4/assignment_loss‚2.5961993063489595 2loss/SRqqt1/assignment_loss‚2.7809580355077177 2loss/SRqqt2/assignment_loss‚2.2887412151021342 0loss/FBt2/assignment_loss‚0.5076767024066713 /loss/FRt1/detection_loss‚2.7704896503373195 .loss/FRt3/detection_loss‚2.615875117088619 -loss/FRt4/detection_loss‚1.80060393223539 /loss/FRt1/assignment_loss‚4.866056511276646 /loss/FRt3/assignment_loss‚4.359389069833253 1loss/SRqqt4/assignment_loss‚1.704538309574127 1loss/SRqqt4/detection_loss‚1.1107746445215665 .loss/FBt2/detection_loss‚2.819126102659437 % _timestamp‚1.7883996453829417e+09 _runtime‚4708.942707903 _step‚360 0loss/FBt3/assignment_loss‚0.4059319499776595 epoch‚3 1loss/SRqqt3/assignment_loss‚2.394083340962728 -loss/FBt4/detection_loss‚0.45725758143879ó‡ßY¡Ó 0loss/FBt2/assignment_loss‚0.5076767024066713 /loss/FRt1/detection_loss‚2.7704896503373195 .loss/FRt3/detection_loss‚2.615875117088619 -loss/FRt4/detection_loss‚1.80060393223539 /loss/FRt1/assignment_loss‚4.866056511276646 /loss/FRt3/assignment_loss‚4.359389069833253 1loss/SRqqt4/assignment_loss‚1.704538309574127 1loss/SRqqt4/detection_loss‚1.1107746445215665 .loss/FBt2/detection_loss‚2.819126102659437 % _timestamp‚1.7883996453829417e+09 _runtime‚4708.942707903 _step‚360 0loss/FBt3/assignment_loss‚0.4059319499776595 epoch‚3 1loss/SRqqt3/assignment_loss‚2.394083340962728 -loss/FBt4/detection_loss‚0.45725758143879 1loss/SRqqt3/detection_loss‚2.3012915501991906 trainer/global_step‚4399 /loss/FRt2/assignment_loss‚4.619996865590413 1loss/FBt4/assignment_loss‚0.27741776494768755 0loss/FBt1/assignment_loss‚0.3319794340215075 1loss/SRqqt1/detection_loss‚2.2962625332780786 1loss/SRqqt2/detection_loss‚1.8834795432706033 /loss/FBt1/detection_loss‚3.5785135626792908 /loss/FBt3/detection_loss‚2.7424696021609836 %loss/total_loss‚54.05738419771303 .loss/FRt2/detection_loss‚2.548271872294255 0loss/FRt4/assignment_loss‚2.5961993063489595 2loss/SRqqt1/assignment_loss‚2.7809580355077177 2loss/SRqqt2/assignment_loss‚2.2887412151021342èf^A2¢‚(:… ž ãÔð¢“Ó gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚80.258' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.0516 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›©ú@£‚(:µ ž ãÔçÒáÕ disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875 disk.dm-0.out‚ 52561.734375$ memory_percent‚98.92535209655762 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.sda.in‚ 0.07421875% disk./.usageGB‚1053.9645195007324 disk.drbd1777.out‚0 disk.dm-0.in‚ 670.16796875 network.sent‚ 1.0489592e+07 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 670.16796875 disk.sda2.in‚0 cpu‚25.25203741365818 network.recv‚ 1.658465e+06 disk.sda1.in‚0 disk.rbd1.out‚ 2.20947265625 disk.rbd0.out‚0! disk.nvme0n1.out‚ 52561.734375 disk.sda.out‚ 3106.27734375 proc.cpu.threads‚17 disk.rbd2.out‚ 140.4921875 disk.sda2.out‚0 disk.sda3.out‚ 3106.27734375 disk.rbd0.in‚0* proc.memory.percent‚17.372679710388184) disk./.usagePercent‚70.73244323882874# disk.nvme0c0n1.out‚ 52561.734375 disk.dm-1.out‚ 3106.27734375' proc.memory.availableMB‚ 176.0703125# proc.memory.rssMB‚ 2846.33984375æ⑤‚(:† ­ ãÔîëÒ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.488' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4976 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0— ¯yJ¥‚(:¿ ­ ãÔÁà¬Õ network.recv‚ 1.664211e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 142.08203125# memory_percent‚98.9255428314209 disk.sda.in‚ 0.07421875 disk.nvme0n1.in‚ 670.16796875' proc.memory.availableMB‚ 176.0390625% disk./.usageGB‚1053.9645538330078 disk.sda1.in‚0 disk.dm-1.out‚ 3106.703125) disk./.usagePercent‚70.73244554289654 disk.rbd3.out‚0( disk.nvme0c0n1.out‚52562.23095703125 disk.dm-0.in‚ 670.16796875 cpu‚25.251298915038944 proc.cpu.threads‚17 disk.sda.out‚ 3106.703125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3106.703125 disk.rbd1.out‚ 2.29541015625 disk.dm-1.in‚ 0.07421875 network.sent‚ 1.0597594e+07 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚52562.23095703125# disk.dm-0.out‚52562.23095703125 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 670.16796875# proc.memory.rssMB‚ 2846.33984375 disk.rbd0.in‚0* proc.memory.percent‚17.372679710388184²Q¯Äœ¦‚(:‘ ¼ ãÔˆ¤ˆÔ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.451' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.490199999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆË¾Q§‚(:Æ ¼ ãÔ„ŒÖ disk.drbd1777.out‚0 disk.rbd1.out‚ 2.35791015625 disk.dm-1.out‚ 3107.41015625& disk.nvme0n1.out‚52562.69287109375 disk.sda1.in‚0$ memory_percent‚98.92399311065674* proc.memory.percent‚17.372679710388184 disk.rbd3.out‚0( disk.nvme0c0n1.out‚52562.69287109375 disk.dm-0.in‚ 670.16796875 disk.sda.in‚ 0.07421875 cpu‚25.25070966101828 disk.drbd1777.in‚0 disk.sda3.out‚ 3107.41015625 disk.sda.out‚ 3107.41015625 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.0603865e+07 network.recv‚ 1.668554e+06% disk./.usageGB‚1053.9644966125488 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.16796875) disk./.usagePercent‚70.73244170278352 disk.rbd3.in‚0# proc.memory.rssMB‚ 2846.33984375 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875( proc.memory.availableMB‚ 176.29296875 disk.rbd2.in‚0 disk.rbd2.out‚ 142.14453125" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd0.in‚0# disk.dm-0.out‚52562.69287109375ùÀ/6š¨‚(: Ë ãÔ„éÛÒ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.54' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.508000000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã²+DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160235654317056 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201709029 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170365691377152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291573760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198353289680896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160235637277696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8116262060544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83804493 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117904538 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198353313620992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170365709380096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8116262060544 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 trainer/global_step‚4449 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8116028474880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8091782266634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49151536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170365709380096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206559526668800 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117930720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8206504621568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168327436583690 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8116028474880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206559818242560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8206236987904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚178481719852032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 $ _timestamp‚1.788399691678703e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170365691377152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117904538 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31862074 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81012876 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83804493 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚178481719852032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31861342 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83804491 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168327190574478 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201735213 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267633664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83804491 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 _runtime‚ 4755.23874018 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚178481971440640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚178481971440640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49151534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8091553296782¿â½Æ‹¯ÝO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198353313620992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170365709380096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8116262060544 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 trainer/global_step‚4449 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8116028474880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8091782266634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49151536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170365709380096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206559526668800 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117930720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8206504621568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168327436583690 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8116028474880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206559818242560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8206236987904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚178481719852032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 $ _timestamp‚1.788399691678703e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170365691377152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117904538 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31862074 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81012876 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83804493 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 UADeviceStatsMonitor.on_tr¸HóZXain_batch_start/allocated_bytes.all.freed‚178481719852032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31861342 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83804491 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168327190574478 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201735213 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 267633664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83804491 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 _runtime‚ 4755.23874018 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚178481971440640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚178481971440640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49151534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8091553296782 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚734 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160235654317056 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201709029 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170365691377152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 291573760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198353289680896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160235637277696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8116262060544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83804493 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117904538ê…Ÿ2°- _wandbruntime‚4756  _runtime‚4756áê&±äM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚178520922892288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚178520671290368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83822670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31868807 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83822672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170402833667072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81030285 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117930613 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117956821 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198396692530688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160270581010432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168364167010714 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201779493 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198396668590592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8117837623296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8208329904128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201779493 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206604730874368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201753283 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8093357030222 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 % _timestamp‚1.7883996925701022e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170402851670016 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 trainer/global_step‚4449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49162187 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206605022434816 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚178520922892288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚178520671290368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83822670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31868807 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83822672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170402833667072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81030285 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117930613 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117956821 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198396692530688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160270581010432 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168364167010714 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201779493 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198396668590592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8117837623296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8208329904128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201779493 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206604730874368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201753283 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8093357030222 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 % _timestamp‚1.7883996925701022e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170402851670016 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 trainer/global_step‚4449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49162187 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354ëñdœÕ2³- _wandbruntime‚4756  _runtime‚4756ÍÌùU´Ï /loss/FRt2/assignment_loss‚4.126054914224715 0loss/FBt1/assignment_loss‚0.6779783378026423 /loss/FBt3/assignment_loss‚0.614230726246855 0loss/FBt4/assignment_loss‚0.2556551395947546 /loss/FRt2/detection_loss‚2.3748448271126974 -loss/FRt3/detection_loss‚2.59983141525932 /loss/FBt2/detection_loss‚3.1504812240600586 % _timestamp‚1.7883996925724795e+09 1loss/SRqqt2/detection_loss‚1.7613141364172886 /loss/FRt1/assignment_loss‚4.528165419896443 /loss/FRt3/assignment_loss‚5.396677535513173 0loss/FBt2/assignment_loss‚0.6810695550135019 /loss/FRt4/detection_loss‚1.7984431104527578 .loss/FBt3/detection_loss‚2.634819371359689 epoch‚3 2loss/SRqqt1/assignment_loss‚2.3635052389958324 1loss/SRqqt2/assignment_loss‚2.210909370790448 2loss/SRqqt3/assignment_loss‚2.7870700720584756 .loss/FRt1/detection_loss‚2.335104469742094 1loss/SRqqt3/detection_loss‚2.2517959674199424 /loss/FRt4/assignment_loss‚2.238999491488492 2loss/SRqqt4/assignment_loss‚1.8153616093331948 %loss/total_loss‚53.01033674238805 trainer/global_step‚4449 1loss/SRqqt4/detection_loss‚0.9599567506229505 /loss/FBt1/detection_loss‚2.3816637992858887 /loss/FBt4/detection_loss‚0.6723351753470335 _runtime‚4756.132219072 _step‚364 1loss/SRqqt1/detection_loss‚2.3940690843498005ìzw-ZµÔ .loss/FRt1/detection_loss‚2.335104469742094 1loss/SRqqt3/detection_loss‚2.2517959674199424 /loss/FRt4/assignment_loss‚2.238999491488492 2loss/SRqqt4/assignment_loss‚1.8153616093331948 %loss/total_loss‚53.01033674238805 trainer/global_step‚4449 1loss/SRqqt4/detection_loss‚0.9599567506229505 /loss/FBt1/detection_loss‚2.3816637992858887 /loss/FBt4/detection_loss‚0.6723351753470335 _runtime‚4756.132219072 _step‚364 1loss/SRqqt1/detection_loss‚2.3940690843498005 /loss/FRt2/assignment_loss‚4.126054914224715 0loss/FBt1/assignment_loss‚0.6779783378026423 /loss/FBt3/assignment_loss‚0.614230726246855 0loss/FBt4/assignment_loss‚0.2556551395947546 /loss/FRt2/detection_loss‚2.3748448271126974 -loss/FRt3/detection_loss‚2.59983141525932 /loss/FBt2/detection_loss‚3.1504812240600586 % _timestamp‚1.7883996925724795e+09 1loss/SRqqt2/detection_loss‚1.7613141364172886 /loss/FRt1/assignment_loss‚4.528165419896443 /loss/FRt3/assignment_loss‚5.396677535513173 0loss/FBt2/assignment_loss‚0.6810695550135019 /loss/FRt4/detection_loss‚1.7984431104527578 .loss/FBt3/detection_loss‚2.634819371359689 epoch‚3 2loss/SRqqt1/assignment_loss‚2.3635052389958324 1loss/SRqqt2/assignment_loss‚2.210909370790448 2loss/SRqqt3/assignment_loss‚2.7870700720584756ìdKÈÒœ¶‚(:‘ Ú ãÔ“ãÐÓ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.438' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.087600000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»(¢šO·‚(:Ä Ú ãÔ¡ˆÓÕ disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 3108.71484375 disk.rbd1.out‚ 2.39013671875' disk.nvme0c0n1.out‚52662.6728515625 disk.sda.in‚ 0.07421875# proc.memory.rssMB‚ 2846.33984375 network.sent‚ 1.061541e+07 network.recv‚ 1.676638e+06 disk.dm-1.in‚ 0.07421875% disk.nvme0n1.out‚52662.6728515625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.372679710388184 disk.dm-1.out‚ 3108.71484375$ memory_percent‚98.92427921295166 disk.rbd2.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚52662.6728515625( proc.memory.availableMB‚ 176.24609375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 3108.71484375 disk.sda1.in‚0! disk.rbd2.out‚142.30908203125 disk.sda3.in‚ 0.07421875 disk.nvme0n1.in‚ 670.16796875 cpu‚25.24945920534348 proc.cpu.threads‚17( disk./.usagePercent‚70.7324527111075% disk./.usageGB‚1053.9646606445312 disk.drbd1777.in‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.dm-0.in‚ 670.16796875°ÒD¸‚(:„ é ãÔÚ¾‘Ó gpu.0.gpu‚40.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.03' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.006 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k/kM¹‚(: é ãÔ³êïÕ disk.rbd1.in‚0 disk.dm-1.out‚ 3109.109375 disk.rbd0.in‚0' proc.memory.availableMB‚ 176.2265625 disk.sda.out‚ 3109.109375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.372679710388184 network.sent‚ 1.0723369e+07 disk.nvme0n1.in‚ 670.16796875 disk.dm-0.in‚ 670.16796875 disk.dm-1.in‚ 0.07421875$ memory_percent‚98.92439842224121# proc.memory.rssMB‚ 2846.33984375! disk.rbd2.out‚142.32470703125( disk.nvme0c0n1.out‚52761.87158203125# disk.dm-0.out‚52761.87158203125 disk.sda1.out‚0 disk.rbd1.out‚ 2.39013671875( disk./.usagePercent‚70.7324511750623& disk.nvme0n1.out‚52761.87158203125 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3109.109375" disk.nvme0c0n1.in‚ 670.16796875 network.recv‚ 1.682332e+06% disk./.usageGB‚1053.9646377563477 proc.cpu.threads‚17 disk.sda1.in‚0 cpu‚25.248834980358808 disk.rbd2.in‚0 disk.sda2.in‚0L„… œº‚(:‘ ø ãÔþ¦õÒ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.429' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.285800000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q½„P»‚(:Å ø ãÔÔ®‘Õ disk.sda.out‚ 3109.76953125! disk.rbd2.out‚143.96142578125* proc.memory.percent‚17.372679710388184 cpu‚25.248225090724397 disk.rbd2.in‚0 disk.sda3.out‚ 3109.76953125 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 670.16796875 disk.nvme0n1.in‚ 670.16796875 disk.drbd1777.in‚0( proc.memory.availableMB‚ 175.99609375 proc.cpu.threads‚17 network.recv‚ 1.686798e+06 disk.sda2.in‚0 disk.dm-1.in‚ 0.07421875& disk.nvme0n1.out‚52762.27001953125 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0$ memory_percent‚98.92580509185791% disk./.usageGB‚1053.9645805358887( disk.nvme0c0n1.out‚52762.27001953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 670.16796875# proc.memory.rssMB‚ 2846.33984375 network.sent‚ 1.0729691e+07) disk./.usagePercent‚70.73244733494928 disk.rbd1.in‚0 disk.rbd1.out‚2.453125# disk.dm-0.out‚52762.27001953125 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.out‚ 3109.76953125 disk.sda1.out‚0IË2¼- _wandbruntime‚4802  _runtime‚4802ð)n½¿ % lr-AdamW/pg2‚0.000176378025424884 trainer/global_step‚4499 % _timestamp‚1.7883997390138865`@®=We+09 _runtime‚4802.573711227 _step‚365 % lr-AdamW/pg1‚0.000176378025424884ïv#%Ê¾Ä _step‚365 % lr-AdamW/pg1‚0.000176378025424884 % lr-AdamW/pg2‚0.000176378025424884 trainer/global_step‚4499 % _timestamp‚1.7883997390138865e+09 _runtime‚4802.573711227í=¼šJ2¿- _wandbruntime‚4802  _runtime‚4802êd̆ß'ÀÙO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200522235165696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203925081 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81891733 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81891019 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161981988985856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8181753559990 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 $ _timestamp‚1.788399739014654e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180429557771264 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119237822 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8206500309504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84713441 _step‚366 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49684186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32206835 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8297640641536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170163971515698 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203951265 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180429557771264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8206733895168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200522259105792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180429306182656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8206500309504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8181982529842 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180429306182656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49684184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170163725506486 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172222805873152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8206733895168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376Á|+ä'ÁÞO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180429557771264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8206733895168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200522259105792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180429306182656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8206500309504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8181982529842 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180429306182656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49684184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170163725506486 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172222805873152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8206733895168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200522235165696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203925081 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81891733 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81891019 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161981988985856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8181753559990 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 $ _timestamp‚1.788399739014654e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180429557771264 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119237822 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8206500309504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84713441 _step‚366 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49684186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32206835 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8297640641536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170163971515698 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 203951265 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504îžZk[2Â- _wandbruntime‚4803  _runtime‚4803ÏŽFiê&ÃäM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119263923 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119237715 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8208309457920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚180468509222912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚208865370391552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84731620 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8299471047168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8183557293430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8299732376064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180468509222912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203995545 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200565614075392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84731622 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203995545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4803.492516007 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8208543056896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208865085122560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203969335 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203969335 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261328896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8183786263490 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚367 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81909040 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49694837 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119237715 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81908351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8208543056896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172259948163072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200565638015488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172259966166016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170200701942722 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170200455933302 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162016915679232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172259966166016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32213514 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8208309457920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84731620 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203995545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4803.492516007 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8208543056896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208865085122560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203969335 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203969335 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261328896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8183786263490 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚367 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81909040 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 M>DeviceStatsMonitor.on_train_batÜ=RÙYch_end/requested_bytes.all.peak‚ 6651354441 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49694837 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119237715 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81908351 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8208543056896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172259948163072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200565638015488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172259966166016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170200701942722 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170200455933302 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162016915679232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172259966166016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32213514 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8208309457920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84731620 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180468509222912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203995545 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200565614075392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84731622 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50216836 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚182376892513280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32553046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚4549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32552333 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 % _timestamp‚1.7883997859612591e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82769882 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120518742 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206167317 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211080520301056 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 _step‚370 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _runtime‚4849.521334412 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120544924 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211080237115904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚182377144101888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85622391 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8389056465408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120518742 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚182376892513280 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8297205729792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202691204590592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8296972144128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120544924 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚182377144101888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8389315710464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174079938372096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85622393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174079920369152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8296972144128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8272182793050 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206141133 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163728306615296 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8271953823198 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0a;6#ôÓßO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8297205729792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202691204590592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8296972144128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120544924 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚182377144101888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8389315710464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174079938372096 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85622393 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174079920369152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8296972144128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8272182793050 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206141133 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163728306615296 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8271953823198 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50216836 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚182376892513280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32553046 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚4549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32552333 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 % _timestamp‚1.7883997859612591e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82769882 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120518742 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 LBDevi e×ñ ceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206167317 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211080520301056 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 _step‚370 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 _runtime‚4849.521334412 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120544924 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211080237115904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚182377144101888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85622391 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8389056465408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120518742 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚182376892513280 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82786949 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174117080662016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285268992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8273757556638 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206211597 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206185387 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85640572 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163763233308672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8298781292544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8299014891520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50227487 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120571025 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚182415843951616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120571025 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172037236874730 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32559462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _step‚371 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8273986526698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82787639 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32560150 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261328896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206211597 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 trainer/global_step‚4549 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85640572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883997868448846e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206185387 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163763250348032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8298781292544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4850.405640446 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8390887592960 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8299014891520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚182416095553536 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚182416095553536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211125447153152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172036990865310 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163763233308672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8298781292544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8299014891520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50227487 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120571025 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚182415843951616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120571025 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172037236874730 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32559462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 _step‚371 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8273986526698 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82787639 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32560150 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 261328896 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206211597 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 trainer/global_step‚4549 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85640572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883997868448846e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206185387 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163763250348032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8298781292544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4850.405640446 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8390887592960 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8299014891520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚182416095553536 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚182416095553536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211125447153152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172036990865310 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82786949 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174117080662016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285268992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8273757556638 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206211597 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206185387 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85640572 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360óop‹Ú2×- _wandbruntime‚4850  _runtime‚4850;ä£HYØÓ 2loss/FBt3/assignment_loss‚0.039118631432415896 0loss/SRqqt1/detection_loss‚2.733805621823957 _runtime‚4850.406907012 /loss/FRt1/assignment_loss‚4.774032870064611 0loss/FRt4/assignment_loss‚2.5629513513210207 1loss/SRqqt2/assignment_loss‚2.449876214663188 2loss/SRqqt3/assignment_loss‚2.5278903584222534 0loss/FBt2/assignment_loss‚0.3649931942458021 .loss/FBt1/detection_loss‚3.901877708733082 2loss/SRqqt1/assignment_loss‚2.1648080733514603 /loss/FRt2/detection_loss‚2.4289409197866916 0loss/SRqqt4/detection_loss‚1.175964253804376 /loss/FBt2/detection_loss‚2.3610424887050283 _step‚372 0loss/FRt2/assignment_loss‚3.2957599073648454 1loss/SRqqt2/detection_loss‚1.6475223251183828 1loss/SRqqt3/detection_loss‚2.2751596425030685 .loss/FBt3/detection_loss‚4.479934414227803 /loss/FBt4/detection_loss‚0.9449794532676392 /loss/FRt3/assignment_loss‚4.605223089456558 /loss/FRt3/detection_loss‚2.7727479702896543 epoch‚3 trainer/global_step‚4549 % _timestamp‚1.7883997868471735e+09 2loss/SRqqt4/assignment_loss‚1.5642342598688217 1loss/FBt4/assignment_loss‚0.13762280504498475 .loss/FRt1/detection_loss‚2.571537910596184 %loss/total_loss‚54.25050723609365 0loss/FBt1/assignment_loss‚0.3551480108872056 .loss/FRt4/detection_loss‚2.115335761114608Î)Mü^ÙØ 2loss/SRqqt1/assignment_loss‚2.1648080733514603 /loss/FRt2/detection_loss‚2.4289409197866916 0loss/SRqqt4/detection_loss‚1.175964253804376 /loss/FBt2/detection_loss‚2.3610424887050283 _step‚372 0loss/FRt2/assignment_loss‚3.2957599073648454 1loss/SRqqt2/detection_loss‚1.6475223251183828 1loss/SRqqt3/detection_loss‚2.2751596425030685 .loss/FBt3/detection_loss‚4.479934414227803 /loss/FBt4/detection_loss‚0.9449794532676392 /loss/FRt3/assignment_loss‚4.605223089456558 /loss/FRt3/detection_loss‚2.7727479702896543 epoch‚3 trainer/global_step‚4549 % _timestamp‚1.7883997868471735e+09 2loss/SRqqt4/assignment_loss‚1.5642342598688217 1loss/FBt4/assignment_loss‚0.13762280504498475 .loss/FRt1/detection_loss‚2.571537910596184 %loss/total_loss‚54.25050723609365 0loss/FBt1/assignment_loss‚0.3551480108872056 .loss/FRt4/detection_loss‚2.115335761114608 2loss/FBt3/assignment_loss‚0.039118631432415896 0loss/SRqqt1/detection_loss‚2.733805621823957 _runtime‚4850.406907012 /loss/FRt1/assignment_loss‚4.774032870064611 0loss/FRt4/assignment_loss‚2.5629513513210207 1loss/SRqqt2/assignment_loss‚2.449876214663188 2loss/SRqqt3/assignment_loss‚2.5278903584222534 0loss/FBt2/assignment_loss‚0.3649931942458021 .loss/FBt1/detection_loss‚3.901877708733082ô›u%œÚ‚(:‘ ´¡ãÔ†‹ÿÒ gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.752' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.350399999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nøl)LÛ‚(:Á ´¡ãÔÝdzÖ# proc.memory.rssMB‚ 2846.33984375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.in‚ 670.16796875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 3111.8515625 disk.sda3.in‚ 0.07421875' disk.nvme0c0n1.out‚52962.3916015625 network.recv‚ 1.705223e+06% disk./.usageGB‚1053.9639701843262 disk.rbd3.in‚0* proc.memory.percent‚17.372679710388184 cpu‚25.245726460587235 disk.rbd0.in‚0( disk./.usagePercent‚70.7324063737438 disk.sda3.out‚ 3111.8515625 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0" disk.dm-0.out‚52962.3916015625 disk.sda.out‚ 3111.8515625 disk.nvme0n1.in‚ 670.16796875$ memory_percent‚98.92783164978027 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.rbd2.out‚144.1416015625 network.sent‚ 1.0855854e+07 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda.in‚ 0.07421875" disk.nvme0c0n1.in‚ 670.16796875% disk.nvme0n1.out‚52962.3916015625' proc.memory.availableMB‚ 175.6640625iê×Ü‚(:… áãÔ©¤ÛÒ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚80.258' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.0516 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{Ù†ÕNÝ‚(:à áãÔÐé¯Õ* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17) disk./.usagePercent‚70.73241021385681 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.16796875 network.sent‚ 1.0963499e+07 disk.dm-1.out‚ 3112.42578125" disk.dm-0.out‚52962.8544921875% disk.nvme0n1.out‚52962.8544921875 disk.dm-0.in‚ 670.16796875' proc.memory.availableMB‚ 175.6640625# proc.memory.rssMB‚ 2846.33984375 cpu‚25.24503641877254 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.rbd2.out‚144.1572265625 disk.rbd3.out‚0 disk.rbd0.in‚0$ memory_percent‚98.92783164978027% disk./.usageGB‚1053.9640274047852 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 670.16796875' disk.nvme0c0n1.out‚52962.8544921875 network.recv‚ 1.71077e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 3112.42578125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 3112.42578125 disk.rbd1.out‚ 2.51611328125 disk.dm-1.in‚ 0.07421875N0ú‘Þ‚(:† Ò¡ãÔÚôÎÓ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.038' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í,¹Oß‚(:Ä Ò¡ãÔƒ÷•Ö disk.sda.in‚ 0.07421875 disk.sda.out‚ 3112.9921875 network.sent‚ 1.0968901e+07 disk.rbd0.out‚0 disk.dm-0.in‚ 670.16796875# proc.memory.rssMB‚ 2846.33984375 disk.rbd3.out‚0 disk.sda3.out‚ 3112.9921875 disk.dm-1.out‚ 3112.9921875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 670.16796875 network.recv‚ 1.714479e+06 disk.rbd2.out‚145.7939453125 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 670.16796875 disk.dm-1.in‚ 0.07421875' proc.memory.availableMB‚ 176.1171875* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17) disk./.usagePercent‚70.73261553189928% disk./.usageGB‚1053.9670867919922 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0# memory_percent‚98.9250659942627 disk.rbd2.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚52966.54931640625 disk.rbd1.out‚ 2.51611328125& disk.nvme0n1.out‚52966.54931640625# disk.dm-0.out‚52966.54931640625 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.244432958314274 disk.sda2.out‚0s©®D2à- _wandbruntime‚4896  _runtime‚4896û[¡ÇáÁ & lr-AdamW/pg1‚0.0001737327834977165 & lr-AdamW/pg2‚0.0001737327834977165 trainer/global_step‚4599 % _timestamp‚1.7883998327036433e+09 _runtime‚4896.263441677 _step‚373’S=3ÌâÆ % _timestamp‚1.7883998327036433e+09 _runtime‚4896.263441677 _step‚373 & lr-AdamW/pg1‚0.0001737327834977165 & lr-AdamW/pg2‚0.0001737327834977165 trainer/global_step‚4599õÞ72ã- _wandbruntime‚4896  _runtime‚4896Š’Ø+äÚO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208357185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121825844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚175937034865152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8387443978752 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚735 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86531343 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83649009 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50749484 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213340618634752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8480753841664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173837041379714 L?Deviyœšà"ceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86531341 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50749486 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32900258 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83649744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173836795370502 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184324478843904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165474641284096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚175937052868096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204860126135296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4896.264475545 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175937052868096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86531343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8362154086406 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 % _timestamp‚1.7883998327044108e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173836795370502 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184324478843904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165474641284096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚175937052868096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204860126135296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4896.264475545 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175937052868096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86531343 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8362154086406 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 % _timestamp‚1.7883998327044108e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033öKLüÃ2æ- _wandbruntime‚4897  _runtime‚4897¹Sé&çãM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175974195158016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86549522 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208427649 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204903528985088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 _runtime‚4897.144178892 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175974177155072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8389253127168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8364186789906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8389486726144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208401439 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165509567977472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8389486726144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8389253127168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213386112483840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204903505044992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175974195158016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184363681884160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86549522 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86549520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165509585016832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 $ _timestamp‚1.788399833583735e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50760137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173873525797318 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208427649 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚175974177155072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32906882 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173873771806738 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121878127 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121878127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8482583498752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208401439 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184363681884160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83666307 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8482315878400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121851919 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213385820923392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8363957819846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4599 ^<±ý èèM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86549522 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86549520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165509585016832 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 $ _timestamp‚1.788399833583735e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50760137 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173873525797318 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208427649 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚175974177155072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32906882 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173873771806738 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121878127 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121878127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 291560448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8482583498752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208401439 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 T9öñF@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184363681884160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83666307 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8482315878400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121851919 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213385820923392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8363957819846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4599 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175974195158016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86549522 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208427649 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204903528985088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 _runtime‚4897.144178892 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175974177155072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8389253127168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8364186789906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8389486726144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208401439 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 267620352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165509567977472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8389486726144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8389253127168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213386112483840 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204903505044992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175974195158016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚184363681884160÷®àqx2é- _wandbruntime‚4897  _runtime‚4897DÔæŽUêÏ /loss/FBt1/detection_loss‚3.0291555523872375 /loss/FBt2/detection_loss‚2.4154017567634583 %loss/total_loss‚52.68133632310743 % _timestamp‚1.7883998335858426e+09 _step‚376 /loss/FRt3/assignment_loss‚4.596069196859996 .loss/FRt1/detection_loss‚2.300388809369535 1loss/SRqqt1/detection_loss‚1.9979715678426955 1loss/SRqqt3/detection_loss‚2.1944584156337537 1loss/SRqqt4/detection_loss‚1.6087145915627479 /loss/FRt2/detection_loss‚2.4546653323703342 .loss/FBt3/detection_loss‚3.134000410636266 trainer/global_step‚4599 .loss/FRt4/detection_loss‚2.154156943783164 1loss/SRqqt2/detection_loss‚1.8914274719723485 /loss/SRqqt2/assignment_loss‚2.4406714810614 /loss/FBt4/detection_loss‚0.6857618565360705 epoch‚3 /loss/FRt1/assignment_loss‚4.666263872263383 0loss/FRt4/assignment_loss‚1.7995874546468258 1loss/FBt1/assignment_loss‚0.47098096429544967 0loss/FBt2/assignment_loss‚0.5283428906248167 0loss/FBt3/assignment_loss‚0.6177046255518993 _runtime‚4897.145957883 /loss/FRt2/assignment_loss‚4.511667595969306 1loss/SRqqt1/assignment_loss‚2.278710440794627 2loss/SRqqt3/assignment_loss‚2.4254894413446126 2loss/SRqqt4/assignment_loss‚1.7241554163396358 0loss/FBt4/assignment_loss‚0.1765760539354283 .loss/FRt3/detection_loss‚2.579014180562435‹² ZëÔ /loss/FRt1/assignment_loss‚4.666263872263383 0loss/FRt4/assignment_loss‚1.7995874546468258 1loss/FBt1/assignment_loss‚0.47098096429544967 0loss/FBt2/assignment_loss‚0.5283428906248167 0loss/FBt3/assignment_loss‚0.6177046255518993 _runtime‚4897.145957883 /loss/FRt2/assignment_loss‚4.511667595969306 1loss/SRqqt1/assignment_loss‚2.278710440794627 2loss/SRqqt3/assignment_loss‚2.4254894413446126 2loss/SRqqt4/assignment_loss‚1.7241554163396358 0loss/FBt4/assignment_loss‚0.1765760539354283 .loss/FRt3/detection_loss‚2.579014180562435 /loss/FBt1/detection_loss‚3.0291555523872375 /loss/FBt2/detection_loss‚2.4154017567634583 %loss/total_loss‚52.68133632310743 % _timestamp‚1.7883998335858426e+09 _step‚376 /loss/FRt3/assignment_loss‚4.596069196859996 .loss/FRt1/detection_loss‚2.300388809369535 1loss/SRqqt1/detection_loss‚1.9979715678426955 1loss/SRqqt3/detection_loss‚2.1944584156337537 1loss/SRqqt4/detection_loss‚1.6087145915627479 /loss/FRt2/detection_loss‚2.4546653323703342 .loss/FBt3/detection_loss‚3.134000410636266 trainer/global_step‚4599 .loss/FRt4/detection_loss‚2.154156943783164 1loss/SRqqt2/detection_loss‚1.8914274719723485 /loss/SRqqt2/assignment_loss‚2.4406714810614 /loss/FBt4/detection_loss‚0.6857618565360705 epoch‚3øøó sšì‚(: á¡ãÔãŽÓ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚85.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.164199999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|åŠÑSí‚(:È á¡ãÔÏÖÕ disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0( disk.nvme0c0n1.out‚53067.25634765625 disk.rbd2.in‚0 disk.rbd2.out‚145.8095703125 disk.rbd3.in‚0 disk.sda3.out‚ 3113.47265625 network.sent‚ 1.097437e+07 disk.dm-1.out‚ 3113.47265625# disk.dm-0.out‚53067.25634765625 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.sda.out‚ 3113.47265625 cpu‚25.243846747308417) disk./.usagePercent‚70.73261937201228 disk.dm-0.in‚ 670.16796875 disk.sda.in‚ 0.07421875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875$ memory_percent‚98.92551898956299 network.recv‚ 1.718285e+06 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0& disk.nvme0n1.out‚53067.25634765625 disk.drbd1777.in‚0( proc.memory.availableMB‚ 176.04296875* proc.memory.percent‚17.372679710388184% disk./.usageGB‚1053.9671440124512" disk.nvme0c0n1.in‚ 670.16796875 disk.rbd0.out‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2846.33984375 disk.nvme0n1.in‚ 670.16796875 disk.drbd1777.out‚0÷Ѥ‘î‚(:† ð¡ãÔÎÈøÒ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.1802 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bù8¾Nï‚(:à ð¡ãÔ犺Õ$ disk.nvme0n1.out‚53067.666015625 disk.rbd1.in‚0& disk.nvme0c0n1.out‚53067.666015625 cpu‚25.243210743369218 network.sent‚ 1.1083259e+07 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0 disk.sda3.in‚ 0.07421875$ memory_percent‚98.92404079437256 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0n1.in‚ 670.16796875! disk.rbd2.out‚145.95849609375 disk.rbd3.out‚0* proc.memory.percent‚17.372679710388184 disk.rbd0.in‚0 disk.sda.out‚ 3114.16015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.07421875 network.recv‚ 1.724887e+06( disk./.usagePercent‚70.7326291002986 disk.sda2.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 2.51611328125" disk.nvme0c0n1.in‚ 670.16796875 disk.dm-0.in‚ 670.16796875! disk.dm-0.out‚53067.666015625 disk.drbd1777.in‚0 disk.sda3.out‚ 3114.16015625 disk.dm-1.out‚ 3114.16015625% disk./.usageGB‚1053.9672889709473 disk.rbd2.in‚0( proc.memory.availableMB‚ 176.28515625# proc.memory.rssMB‚ 2846.33984375á{ŸÖœð‚(:‘ ÿ¡ãÔƒù’Ó gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.094' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.218799999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G2âSñ‚(:È ÿ¡ãÔæ³çÕ disk.dm-1.out‚ 3114.57421875 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚98.92561435699463( proc.memory.availableMB‚ 176.02734375 network.sent‚ 1.1088476e+07 disk.nvme0n1.in‚ 670.16796875 disk.sda1.in‚0# proc.memory.rssMB‚ 2846.33984375& disk.nvme0n1.out‚53169.61181640625 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 3114.57421875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 cpu‚25.242622872959924 disk.dm-1.in‚ 0.07421875 disk.rbd1.out‚ 2.51611328125 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0! disk.rbd2.out‚145.97412109375 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 1.728578e+06# disk./.usageGB‚1053.96732711792" disk.nvme0c0n1.in‚ 670.16796875( disk.nvme0c0n1.out‚53169.61181640625 disk.sda.out‚ 3114.57421875 disk.sda3.in‚ 0.07421875 disk.rbd1.in‚0* proc.memory.percent‚17.372679710388184 disk.dm-0.in‚ 670.16796875# disk.dm-0.out‚53169.61181640625) disk./.usagePercent‚70.73263166037394÷¥I2ò- _wandbruntime‚4943  _runtime‚4943käÏÉóà % _timestamp‚1.7883998798815541e+09 _runtime‚4943.441385846 _step‚377 ' lr-AdamW/pg1‚0.00017239057239494312 ' lr-AdamW/pg2‚0.00017239057239494312 trainer/global_step‚4649Éó—ÓÎôÈ ' lr-AdamW/pg1‚0.00017239057239494312 ' lr-AdamW/pg2‚0.00017239057239494312 trainer/global_step‚4649 % _timestamp‚1.7883998798815541e+09 _runtime‚4943.441385846 _step‚377ùë6ÏS2õ- _wandbruntime‚4943  _runtime‚4943‘‹ðà'öÚO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8571886775808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186272316763136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8572146020864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186272065174528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167220992992256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8478149399040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33245649 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207029071620096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123132946 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167220975952896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8478149399040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177794149361152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7883998798823287e+09 _runtime‚4943.442374048 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123132946 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177794167364096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177794149361152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51282136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215600958395904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175673576311722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852Oñ÷î( ÷ßO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8571886775808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186272316763136 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8572146020864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186272065174528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167220992992256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8478149399040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33245649 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207029071620096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123132946 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167220975952896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8478149399040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177794149361152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7883998798823287e+09 _runtime‚4943.442374048 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123132946 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177794167364096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177794149361152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233585664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51282136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 283185152 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215600958395904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175673576311722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8454387053114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210643701 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚177831291651072 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123185229 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210643701 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84544370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51292787 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51292789 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87458470 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177831309654016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33252270 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84545059 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207072450529792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚379 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33251583 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177831309654016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8573705296896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167255919685632 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8454158083054 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8479724961792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚186311268214784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 % _timestamp‚1.7883998807640717e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8573979208704 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚186311016612864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175710306738746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207072474469888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167255902646272 _runtime‚4944.325027742 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 % _timestamp‚1.7883998807640717e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8573979208704 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚186311016612864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175710306738746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207072474469888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167255902646272 _runtime‚4944.325027742 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8454387053114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210643701 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚177831291651072 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123185229 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210643701 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84544370 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51292787û-†:2û- _wandbruntime‚4944  _runtime‚4944dÚ®UüÏ _runtime‚4944.326401391 _step‚380 /loss/FRt2/assignment_loss‚3.752043542407808 1loss/SRqqt2/assignment_loss‚2.339863685830947 0loss/FBt1/assignment_loss‚0.3572478398840758 /loss/FRt3/detection_loss‚2.9533778205513954 .loss/FBt1/detection_loss‚3.050626665353775 /loss/FBt2/detection_loss‚2.4581396920340404 2loss/SRqqt4/assignment_loss‚2.1001063615515614 0loss/FBt2/assignment_loss‚0.5212825962475368 /loss/FBt3/assignment_loss‚0.567515452275984 .loss/FRt1/detection_loss‚2.522716276347637 epoch‚3 0loss/FRt1/assignment_loss‚4.5923008397221565 0loss/FRt4/assignment_loss‚2.7283719393114247 /loss/FBt4/detection_loss‚0.5755496360815879 %loss/total_loss‚54.09509861649744 trainer/global_step‚4649 /loss/FRt3/assignment_loss‚5.220134869217873 /loss/FRt2/detection_loss‚2.5416074650628224 /loss/FRt4/detection_loss‚1.8113756676514943 0loss/SRqqt2/detection_loss‚1.814046518937234 /loss/FBt3/detection_loss‚3.7247422138849893 % _timestamp‚1.7883998807666597e+09 1loss/SRqqt1/assignment_loss‚2.266506679356098 0loss/SRqqt1/detection_loss‚2.474504460580647 2loss/SRqqt3/assignment_loss‚2.2085712941156492 1loss/FBt4/assignment_loss‚0.24522429132803458 0loss/SRqqt4/detection_loss‚0.896413970036783 0loss/SRqqt3/detection_loss‚2.372828838725885Yv ZýÔ 2loss/SRqqt3/assignment_loss‚2.2085712941156492 1loss/FBt4/assignment_loss‚0.24522429132803458 0loss/SRqqt4/detection_loss‚0.896413970036783 0loss/SRqqt3/detection_loss‚2.372828838725885 _runtime‚4944.326401391 _step‚380 /loss/FRt2/assignment_loss‚3.752043542407808 1loss/SRqqt2/assignment_loss‚2.339863685830947 0loss/FBt1/assignment_loss‚0.3572478398840758 /loss/FRt3/detection_loss‚2.9533778205513954 .loss/FBt1/detection_loss‚3.050626665353775 /loss/FBt2/detection_loss‚2.4581396920340404 2loss/SRqqt4/assignment_loss‚2.1001063615515614 0loss/FBt2/assignment_loss‚0.5212825962475368 /loss/FBt3/assignment_loss‚0.567515452275984 .loss/FRt1/detection_loss‚2.522716276347637 epoch‚3 0loss/FRt1/assignment_loss‚4.5923008397221565 0loss/FRt4/assignment_loss‚2.7283719393114247 /loss/FBt4/detection_loss‚0.5755496360815879 %loss/total_loss‚54.09509861649744 trainer/global_step‚4649 /loss/FRt3/assignment_loss‚5.220134869217873 /loss/FRt2/detection_loss‚2.5416074650628224 /loss/FRt4/detection_loss‚1.8113756676514943 0loss/SRqqt2/detection_loss‚1.814046518937234 /loss/FBt3/detection_loss‚3.7247422138849893 % _timestamp‚1.7883998807666597e+09 1loss/SRqqt1/assignment_loss‚2.266506679356098 0loss/SRqqt1/detection_loss‚2.474504460580647ü×KU‘þ‚(:† Ž¢ãÔåÐõÒ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.739' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5478 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Ø3ëRÿ‚(:Ç Ž¢ãÔ³ÁÔÕ disk.sda1.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚70.73262807626845 disk.rbd0.out‚0 disk.sda.out‚ 3115.4765625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.241993604163735 disk.drbd1777.in‚0 network.sent‚ 1.1094785e+07 network.recv‚ 1.733051e+06% disk./.usageGB‚1053.9672737121582 disk.rbd1.in‚0 disk.nvme0n1.in‚ 670.16796875 disk.dm-0.in‚ 670.16796875# proc.memory.rssMB‚ 2846.33984375 disk.drbd1777.out‚0 disk.dm-1.in‚ 0.07421875 disk.sda2.in‚0( proc.memory.availableMB‚ 176.01171875" disk.nvme0c0n1.in‚ 670.16796875# disk.dm-0.out‚53170.04541015625 disk.sda1.out‚0! disk.rbd2.out‚146.03662109375 disk.sda2.out‚0* proc.memory.percent‚17.372679710388184& disk.nvme0n1.out‚53170.04541015625 disk.sda.in‚ 0.07421875$ memory_percent‚98.92570972442627 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3115.4765625 disk.rbd1.out‚ 2.51611328125( disk.nvme0c0n1.out‚53170.04541015625 disk.dm-1.out‚ 3115.4765625áœâ€‚(:„ ¢ãÔÛ°…Ó gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.974 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f¥ŽO‚(:Ä ¢ãÔðÆ«Õ) disk./.usagePercent‚70.73272663916914 disk.rbd1.in‚0' disk.nvme0c0n1.out‚53181.9638671875! disk.rbd2.out‚147.62646484375$ memory_percent‚98.92513751983643% disk.nvme0n1.out‚53181.9638671875 disk.rbd3.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 176.10546875 disk.sda.out‚ 3117.6640625 disk.rbd3.out‚0 network.sent‚ 1.1202824e+07 network.recv‚ 1.739077e+06% disk./.usageGB‚1053.9687423706055 disk.sda3.out‚ 3117.6640625 disk.dm-1.in‚ 0.07421875 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 3117.6640625 disk.rbd0.out‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.372679710388184 cpu‚25.241420010107994 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2846.33984375 disk.rbd1.out‚ 2.51611328125 disk.dm-0.in‚ 670.17578125" disk.nvme0c0n1.in‚ 670.17578125 disk.nvme0n1.in‚ 670.17578125" disk.dm-0.out‚53181.9638671875 disk.sda.in‚ 0.07421875 disk.sda1.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17eÀ˧›‚‚(: ¬¢ãÔ†‹†Ó gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚86.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.377000000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u®$ÖPƒ‚(:Å ¬¢ãÔ³»çÕ disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚98.92516136169434 disk.rbd1.out‚ 2.51611328125 disk.nvme0n1.in‚ 670.17578125" disk.dm-0.out‚53281.1552734375! disk.rbd2.out‚147.68896484375 network.recv‚ 1.743667e+06 disk.sda3.out‚ 3118.69921875 disk.rbd0.in‚0% disk.nvme0n1.out‚53281.1552734375 disk.sda.out‚ 3118.69921875 disk.drbd1777.out‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 176.1015625 disk.sda2.in‚0' disk.nvme0c0n1.out‚53281.1552734375 disk.dm-1.in‚ 0.07421875# proc.memory.rssMB‚ 2846.33984375 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1209137e+07) disk./.usagePercent‚70.73271691088283 disk.dm-1.out‚ 3118.69921875 disk.dm-0.in‚ 670.17578125% disk./.usageGB‚1053.9685974121094 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 670.17578125 disk.rbd2.in‚0 cpu‚25.24079166794805 disk.drbd1777.in‚0* proc.memory.percent‚17.372679710388184Ò ~ 2„- _wandbruntime‚4990  _runtime‚4990ð?0xÇ…Á trainer/global_step‚4699 % _timestamp‚1.7883999268654947e+09 _runtime‚4990.425349747 _step‚381 & lr-AdamW/pg1‚0.0001710356732515992 & lr-AdamW/pg2‚0.0001710356732515992/Å•Ì†Æ & lr-AdamW/pg1‚0.0001710356732515992 & lr-AdamW/pg2‚0.0001710356732515992 trainer/global_step‚4699 % _timestamp‚1.7883999268654947e+09 _runtime‚4990.425349747 _step‚381ý¦}çF2‡- _wandbruntime‚4990  _runtime‚4990 À6 ˆÙO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚188219903093760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88349243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8542554612822 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212789289 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179651263857152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168967327661056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88349243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217861661846528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33590008 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179651281860096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8542783582674 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 _runtime‚4990.426336988 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8568621233664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88349241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_sta 4¦o©rt/requested_bytes.large_pool.peak‚ 6189583360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚188219903093760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209198017104896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8663361556480 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8568387648000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚382 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85404064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51814786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33589280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124440048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179651281860096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217861378661376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177509865234518 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51814784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124466230 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177510111243730 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168967310621696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 $ _timestamp‚1.788399926866305e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8568621233664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124466230 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8568387648000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124440048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚188219651505152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212815473)ØÂuä'‰ÞO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8568621233664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124466230 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8568387648000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124440048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚188219651505152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212815473 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚188219903093760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88349243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8542554612822 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212789289 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179651263857152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168967327661056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88349243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217861661846528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 259245056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33590008 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179651281860096 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8542783582674 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228969852 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 _runtime‚4990.426336988 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8568621233664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88349241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚188219903093760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209198017104896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8663361556480 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8568387648000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚382 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85404064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51814786 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33589280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124440048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179651281860096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217861378661376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177509865234518 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51814784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124466230 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177510111243730 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168967310621696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 $ _timestamp‚1.788399926866305e+09þô‚Ë*2Š- _wandbruntime‚4991  _runtime‚4991dY–©ê&‹äM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8570430395392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85422247 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177546595661334 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124466123 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88367422 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51825439 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209241419954688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _runtime‚4991.325057304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212833543 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88367420 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124466123 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169002254354432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124492331 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8570196796416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33596106 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚383 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179688406147072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8570430395392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212859753 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8544358346262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179688424150016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179688424150016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88367420 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚188258854545408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179688406147072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85421543 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217906581476864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177546841670754 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212859753 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212833543 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51825437 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33596808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 % _timestamp‚1.7883999277639117e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8570196796416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8665185462272 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8665450985472 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124492331 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚188258854545408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚188258602943488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209241396014592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ÛçÀ4ŒéM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚188258602943488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209241396014592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8570430395392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85422247 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 265523200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246009420 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177546595661334 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124466123 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6505365504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88367422 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51825439 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209241419954688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 _runtime‚4991.325057304 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212833543 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88367420 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124466123 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169002254354432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124492331 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8570196796416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33596106 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚383 SEDeviceStatsMonitor.on_¤IÛ»train_batch_end/requested_bytes.small_pool.peak‚ 517504849 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7035944960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179688406147072 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8570430395392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212859753 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8544358346262 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179688424150016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179688424150016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88367420 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚188258854545408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179688406147072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85421543 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217906581476864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177546841670754 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212859753 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212833543 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51825437 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32033 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33596808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 % _timestamp‚1.7883999277639117e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8570196796416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8665185462272 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8665450985472 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124492331 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚188258854545408ÿC¡02- _wandbruntime‚4991  _runtime‚4991xàUŽÏ 1loss/SRqqt4/detection_loss‚1.1192452750385624 0loss/SRqqt2/detection_loss‚1.937155514474838 /loss/FBt4/detection_loss‚0.8449195251358967 trainer/global_step‚4699 /loss/FRt3/assignment_loss‚4.864910503228505 0loss/FBt1/assignment_loss‚0.5620083684722582 /loss/FRt4/detection_loss‚1.9135032926614468 2loss/SRqqt3/assignment_loss‚2.6751947833432093 /loss/FRt2/detection_loss‚2.7213133203260824 /loss/FRt3/detection_loss‚2.3740854524075985 .loss/FBt1/detection_loss‚3.274034837881724 % _timestamp‚1.7883999277663577e+09 0loss/FBt2/assignment_loss‚0.7483881612618765 .loss/FBt3/detection_loss‚3.667375183105469 epoch‚3 /loss/FRt1/assignment_loss‚4.950818429974949 /loss/FRt4/assignment_loss‚2.579945778617492 0loss/SRqqt3/detection_loss‚2.306953461633788 _step‚384 .loss/FBt2/detection_loss‚4.277556657791138 %loss/total_loss‚56.32985978493491 1loss/FBt4/assignment_loss‚0.21447685911054074 2loss/SRqqt1/assignment_loss‚2.4325605023652317 2loss/SRqqt4/assignment_loss‚1.6974124553268903 0loss/FBt3/assignment_loss‚0.1569269376806915 0loss/SRqqt1/detection_loss‚2.355250783264637 /loss/FRt1/detection_loss‚2.6905610578901626 _runtime‚4991.326088297 .loss/FRt2/assignment_loss‚3.84009084557042 2loss/SRqqt2/assignment_loss‚2.12517179837149953U±ÇZÔ .loss/FRt2/assignment_loss‚3.84009084557042 2loss/SRqqt2/assignment_loss‚2.1251717983714995 1loss/SRqqt4/detection_loss‚1.1192452750385624 0loss/SRqqt2/detection_loss‚1.937155514474838 /loss/FBt4/detection_loss‚0.8449195251358967 trainer/global_step‚4699 /loss/FRt3/assignment_loss‚4.864910503228505 0loss/FBt1/assignment_loss‚0.5620083684722582 /loss/FRt4/detection_loss‚1.9135032926614468 2loss/SRqqt3/assignment_loss‚2.6751947833432093 /loss/FRt2/detection_loss‚2.7213133203260824 /loss/FRt3/detection_loss‚2.3740854524075985 .loss/FBt1/detection_loss‚3.274034837881724 % _timestamp‚1.7883999277663577e+09 0loss/FBt2/assignment_loss‚0.7483881612618765 .loss/FBt3/detection_loss‚3.667375183105469 epoch‚3 /loss/FRt1/assignment_loss‚4.950818429974949 /loss/FRt4/assignment_loss‚2.579945778617492 0loss/SRqqt3/detection_loss‚2.306953461633788 _step‚384 .loss/FBt2/detection_loss‚4.277556657791138 %loss/total_loss‚56.32985978493491 1loss/FBt4/assignment_loss‚0.21447685911054074 2loss/SRqqt1/assignment_loss‚2.4325605023652317 2loss/SRqqt4/assignment_loss‚1.6974124553268903 0loss/FBt3/assignment_loss‚0.1569269376806915 0loss/SRqqt1/detection_loss‚2.355250783264637 /loss/FRt1/detection_loss‚2.6905610578901626 _runtime‚4991.326088297€1¸>R‘‚(:† »¢ãÔ»íŠÓ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 8ÀN‘‚(:à »¢ãÔ‘çÐÖ network.sent‚ 1.1214434e+07 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.07421875# proc.memory.rssMB‚ 2846.33984375 disk.sda.in‚ 0.07421875 disk.rbd2.in‚0! disk.rbd2.out‚147.70068359375 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125* proc.memory.percent‚17.372679710388184 proc.cpu.threads‚17$ disk.nvme0n1.out‚53288.392578125 disk.dm-0.in‚ 670.17578125 disk.sda.out‚ 3119.38671875$ memory_percent‚98.92776012420654( proc.memory.availableMB‚ 175.67578125( disk./.usagePercent‚70.7327921770979% disk./.usageGB‚1053.9697189331055 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 670.17578125 disk.sda3.in‚ 0.07421875 disk.sda1.out‚0 network.recv‚ 1.747416e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 3119.38671875 disk.sda3.out‚ 3119.38671875" disk.nvme0c0n1.in‚ 670.17578125 cpu‚25.240213172510643! disk.dm-0.out‚53288.392578125 disk.rbd3.out‚0& disk.nvme0c0n1.out‚53288.392578125Ûiúæ›’‚(: Ê¢ãÔ„ËÞÒ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.261799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0égqG“‚(:¼ Ê¢ãÔÑѽÕ network.recv‚ 1.753736e+06) disk./.usagePercent‚70.73275454399037 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 2.51611328125 disk.drbd1777.out‚0 network.sent‚ 1.1323171e+07 disk.rbd3.out‚0& disk.nvme0c0n1.out‚53387.599609375 disk.dm-1.in‚ 0.07421875 disk.sda.in‚ 0.07421875 disk.sda1.in‚0# proc.memory.rssMB‚ 2846.33984375 cpu‚25.239609032539448% disk./.usageGB‚1053.9691581726074 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 670.17578125$ disk.nvme0n1.out‚53387.599609375 disk.dm-0.in‚ 670.17578125 disk.sda.out‚ 3120.484375 disk.drbd1777.in‚0$ memory_percent‚98.92637729644775 proc.cpu.threads‚17 disk.rbd2.out‚ 147.873046875 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3120.484375 disk.dm-1.out‚ 3120.484375 disk.rbd0.out‚0 disk.rbd0.in‚0! disk.dm-0.out‚53387.599609375( proc.memory.availableMB‚ 175.90234375* proc.memory.percent‚17.372679710388184 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 670.17578125óP~NŽ”‚(:ƒ Ù¢ãÔµÉ€Ó gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.55' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.31 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž^^EF•‚(:» Ù¢ãÔ…“²Õ disk.dm-0.in‚ 670.203125 disk.rbd2.in‚0# proc.memory.rssMB‚ 2846.33984375 disk.sda.in‚ 0.07421875 disk.sda.out‚ 3121.015625 disk.sda1.in‚0 disk.drbd1777.in‚0 cpu‚25.23904694569949) disk./.usagePercent‚70.76644487948886 disk.nvme0n1.in‚ 670.203125 network.sent‚ 1.1328741e+07 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 3121.015625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.nvme0c0n1.in‚ 670.203125 disk.dm-1.out‚ 3121.015625 disk.rbd0.in‚0& disk.nvme0n1.out‚53392.93310546875 disk.sda1.out‚0 disk.rbd2.out‚ 147.888671875% disk./.usageGB‚1054.4711685180664 disk.sda3.in‚ 0.07421875$ memory_percent‚98.92942905426025( proc.memory.availableMB‚ 175.40234375( disk.nvme0c0n1.out‚53392.93310546875 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0# disk.dm-0.out‚53392.93310546875* proc.memory.percent‚17.372679710388184 network.recv‚ 1.758092e+06 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚07Ĭ{2–- _wandbruntime‚5038  _runtime‚5038êgÁÈ—Â ' lr-AdamW/pg1‚0.00016966836842663538 ' lr-AdamW/pg2‚0.00016966836842663538 trainer/global_step‚4749 $ _timestamp‚1.788399974698147e+09 _runtime‚5038.257980465 _step‚385±Ns˜Í˜Ç ' lr-AdamW/pg2‚0.00016966836842663538 trainer/global_step‚4749 $ _timestamp‚1.788399974698147e+09 _runtime‚5038.257980465 _step‚385 ' lr-AdamW/pg1‚0.00016966836842663538Òh!À2™- _wandbruntime‚5038  _runtime‚5038OîÞ'šØO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89258191 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 _step‚386 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190167489424384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8658859482624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190167237835776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293670912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8754783111168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215005341 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89258193 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86283030 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788399974698913e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211366986529792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179346400166526 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215031525 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52347434 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179346646175738 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8632754876030 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125747150 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8658859482624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86283806 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220122039371776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181508378353152 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚4749 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52347436 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170713645290496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190167237835776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89258191 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89258193 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170713662329856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181508378353152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8659093068288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8659093068288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220121745700864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 5038.25894344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125773332 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211366962589696 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8632983845882 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516vîýˆ&›ÝO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220121745700864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚ 5038.25894344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125773332 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211366962589696 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8632983845882 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89258191 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26182 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 _step‚386 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190167489424384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6874205696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8658859482624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚307 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190167237835776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 293670912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8754783111168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7035944960 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215005341 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89258193 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86283030 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788399974698913e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211366986529792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179346400166526 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215031525 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52347434 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179346646175738 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8632754876030 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125747150 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8658859482624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 522164224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86283806 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220122039371776 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181508378353152 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 trainer/global_step‚4749 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52347436 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170713645290496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190167237835776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89258191 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6874205696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32033 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89258193 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6408446976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚32 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170713662329856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitod½ž|[r.on_train_batch_start/active_bytes.large_pool.freed‚181508378353152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8659093068288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8659093068288‚žúê 2œ- _wandbruntime‚5039  _runtime‚5039±÷Ìê&äM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 276880384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211410341499392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170748571983872 % _timestamp‚1.7883999756106174e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181545538646016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8756869672448 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181545538646016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8660902230016 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170748589023232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89276372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125799433 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190206440876032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52358087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 _runtime‚5039.171267982 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190206440876032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179383130593342 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52358089 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181545520643072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220166958231552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8634787579530 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215049595 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 252940288 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86300489 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89276370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215075805 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8660668631040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33942402 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8660902230016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190206189274112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215075805 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8756616732160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125799433 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211410365439488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8634558609470 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208ñu9ï&žéM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181545538646016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6408446976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8660902230016 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170748589023232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89276372 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125799433 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190206440876032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52358087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 _runtime‚5039.171267982 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190206440876032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179383130593342 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52358089 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181545520643072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220166958231552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8634787579530 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215049595 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 522164224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 252940288 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86300489 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6874205696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 530579456 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89276370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215075805 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8660668631040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33942402 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 530579456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8660902230016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190206189274112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215075805 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8756616732160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125799433 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1382 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚732 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211410365439488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8634558609470 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 276880384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7035944960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211410341499392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚307 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170748571983872 % _timestamp‚1.7883999756106174e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181545538646016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8756869672448ƒêsG2Ÿ- _wandbruntime‚5039  _runtime‚5039*öyU Ï epoch‚3 % _timestamp‚1.7883999756130288e+09 2loss/SRqqt4/assignment_loss‚1.8066946868713085 0loss/FBt2/assignment_loss‚0.6406480473597185 .loss/FRt4/detection_loss‚1.928335065521845 1loss/SRqqt1/detection_loss‚2.5239020215498433 /loss/FBt4/detection_loss‚0.6457614367725429 _step‚388 /loss/FRt4/assignment_loss‚2.980964465839107 2loss/SRqqt1/assignment_loss‚1.9895964756205275 1loss/SRqqt2/assignment_loss‚2.463711399472121 /loss/FBt2/detection_loss‚3.6457920244761874 trainer/global_step‚4749 _runtime‚5039.172777847 0loss/FBt1/assignment_loss‚1.0254131663929333 .loss/FBt1/detection_loss‚2.403718872503801 /loss/FRt3/assignment_loss‚4.717334723472595 1loss/FBt4/assignment_loss‚0.21221240353881493 .loss/FRt1/detection_loss‚2.299876720837827 /loss/FRt1/assignment_loss‚4.610881409555112 0loss/FBt3/assignment_loss‚0.2846657828621793 1loss/SRqqt3/detection_loss‚2.4667214062064886 1loss/SRqqt4/detection_loss‚1.1113624579631365 %loss/total_loss‚54.21865151201037 1loss/SRqqt3/assignment_loss‚2.179512483999133 1loss/SRqqt2/detection_loss‚1.6654329083182595 /loss/FRt2/detection_loss‚2.5144822779628964 .loss/FRt3/detection_loss‚2.732647183537483 /loss/FRt2/assignment_loss‚3.837021049526003 .loss/FBt3/detection_loss‚3.531963041850499ÓUZ¡Ô /loss/FBt2/detection_loss‚3.6457920244761874 trainer/global_step‚4749 _runtime‚5039.172777847 0loss/FBt1/assignment_loss‚1.0254131663929333 .loss/FBt1/detection_loss‚2.403718872503801 /loss/FRt3/assignment_loss‚4.717334723472595 1loss/FBt4/assignment_loss‚0.21221240353881493 .loss/FRt1/detection_loss‚2.299876720837827 /loss/FRt1/assignment_loss‚4.610881409555112 0loss/FBt3/assignment_loss‚0.2846657828621793 1loss/SRqqt3/detection_loss‚2.4667214062064886 1loss/SRqqt4/detection_loss‚1.1113624579631365 %loss/total_loss‚54.21865151201037 1loss/SRqqt3/assignment_loss‚2.179512483999133 1loss/SRqqt2/detection_loss‚1.6654329083182595 /loss/FRt2/detection_loss‚2.5144822779628964 .loss/FRt3/detection_loss‚2.732647183537483 /loss/FRt2/assignment_loss‚3.837021049526003 .loss/FBt3/detection_loss‚3.531963041850499 epoch‚3 % _timestamp‚1.7883999756130288e+09 2loss/SRqqt4/assignment_loss‚1.8066946868713085 0loss/FBt2/assignment_loss‚0.6406480473597185 .loss/FRt4/detection_loss‚1.928335065521845 1loss/SRqqt1/detection_loss‚2.5239020215498433 /loss/FBt4/detection_loss‚0.6457614367725429 _step‚388 /loss/FRt4/assignment_loss‚2.980964465839107 2loss/SRqqt1/assignment_loss‚1.9895964756205275 1loss/SRqqt2/assignment_loss‚2.463711399472121„¿®>Ø‘¢‚(:† è¢ãÔ£ÿ÷Ó gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.142' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6284 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äUð6M£‚(: è¢ãÔ¾“Õ) disk./.usagePercent‚70.82150109177996 disk.rbd2.out‚ 148.013671875 disk.rbd3.out‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 673.25390625 disk.dm-1.out‚ 3121.9453125 disk.sda.out‚ 3121.9453125$ memory_percent‚98.92802238464355 disk.drbd1777.in‚0 disk.dm-0.in‚ 673.25390625' proc.memory.availableMB‚ 175.6328125# proc.memory.rssMB‚ 2846.33984375 cpu‚25.238489763038043 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.recv‚ 1.762497e+06 disk.rbd2.in‚0 disk.sda3.out‚ 3121.9453125( disk.nvme0c0n1.out‚54752.75537109375 disk.rbd0.in‚0# disk./.usageGB‚1055.29154586792 disk.rbd3.in‚0 disk.dm-1.in‚ 0.07421875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 673.25390625# disk.dm-0.out‚54752.75537109375 disk.sda.in‚ 0.07421875* proc.memory.percent‚17.372679710388184 network.sent‚ 1.1335057e+07 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.51611328125& disk.nvme0n1.out‚54752.75537109375 proc.cpu.threads‚17ÕïBà‘¤‚(:† ÷¢ãÔšË÷Ò gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.254' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2508 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³(þG¥‚(:¼ ÷¢ãÔé‘ÁÕ disk.rbd2.out‚ 149.603515625 disk.sda3.in‚ 0.07421875& disk.nvme0c0n1.out‚54859.279296875$ disk.nvme0n1.out‚54859.279296875* proc.memory.percent‚17.372679710388184 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0$ disk./.usageGB‚1055.294261932373 disk.dm-1.out‚ 3122.82421875 disk.sda1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚70.82168336914435 disk.sda2.in‚0 disk.rbd1.out‚ 2.51611328125! disk.nvme0c0n1.in‚ 677.1171875 disk.rbd0.out‚0 disk.dm-0.in‚ 677.1171875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚54859.279296875 disk.sda.out‚ 3122.82421875& proc.memory.availableMB‚ 175.609375 network.sent‚ 1.1442789e+07 network.recv‚ 1.768384e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚ 677.1171875 disk.sda.in‚ 0.07421875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 3122.82421875 disk.dm-1.in‚ 0.07421875$ memory_percent‚98.92816543579102# proc.memory.rssMB‚ 2846.33984375 cpu‚25.237929650747752Rìm¦‚(:„ †£ãÔ‡³ýÒ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.79' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.358 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©óÏS§‚(:È †£ãÔ±ÚâÕ& disk.nvme0n1.out‚54860.18701171875# disk.dm-0.out‚54860.18701171875 disk.sda3.out‚ 3123.63671875$ memory_percent‚98.92809391021729* proc.memory.percent‚17.372679710388184 disk.nvme0n1.in‚ 677.44140625 disk.rbd3.in‚0 network.sent‚ 1.1449164e+07( disk.nvme0c0n1.out‚54860.18701171875 disk.rbd3.out‚0 disk.rbd1.out‚ 2.51611328125 cpu‚25.237373856446204 disk.sda3.in‚ 0.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.rbd2.in‚0% disk./.usageGB‚1055.2939796447754" disk.nvme0c0n1.in‚ 677.44140625 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 3123.63671875 disk.dm-0.in‚ 677.44140625 disk.sda2.out‚0( proc.memory.availableMB‚ 175.62109375 network.recv‚ 1.772989e+06) disk./.usagePercent‚70.82166442458681 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2846.33984375 proc.cpu.threads‚17 disk.rbd2.out‚ 149.681640625 disk.sda.out‚ 3123.63671875 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0®Ïœ¨‚(:‘ •£ãÔ ïÓ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÁÈJ©‚(:¿ •£ãÔàöÔÕ network.recv‚ 1.77676e+06 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚98.92516136169434 disk.sda2.out‚0 network.sent‚ 1.1454627e+07 disk.drbd1777.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚54961.5849609375 disk.sda.out‚ 3124.37109375 disk.sda1.in‚0 cpu‚25.23687397407367* proc.memory.percent‚17.372679710388184) disk./.usagePercent‚70.82171690613134 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.sda3.in‚ 0.07421875 disk.rbd1.out‚ 2.51611328125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 678.3515625" disk.dm-0.out‚54961.5849609375# proc.memory.rssMB‚ 2846.33984375 disk.rbd3.out‚0% disk.nvme0n1.out‚54961.5849609375 proc.cpu.threads‚17 disk.rbd2.out‚ 149.697265625! disk.nvme0c0n1.in‚ 678.3515625 disk.sda.in‚ 0.07421875' proc.memory.availableMB‚ 176.1015625 disk.rbd0.out‚0 disk.dm-0.in‚ 678.3515625 disk.sda3.out‚ 3124.37109375% disk./.usageGB‚1055.2947616577148 disk.rbd3.in‚0 disk.dm-1.in‚ 0.07421875 disk.dm-1.out‚ 3124.37109375.Uñ2ª- _wandbruntime‚5085  _runtime‚5085#ŠùJÇ«Á _runtime‚ 5085.94832674 _step‚389 ' lr-AdamW/pg1‚0.00016828894286432784 ' lr-AdamW/pg2‚0.00016828894286432784 trainer/global_step‚4799 $ _timestamp‚1.788400022388526e+092õ—¬Ì¬Æ _step‚389 ' lr-AdamW/pg1‚0.00016828894286432784 ' lr-AdamW/pg2‚0.00016828894286432784 trainer/global_step‚4799 $ _timestamp‚1.788400022388526e+09 _runtime‚ 5085.94832674…Õ”›—2­- _wandbruntime‚5085  _runtime‚5085–2èQ¸ ®ÚO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6651354441 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192115075755008 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1354 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚307 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6408446976 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7035944960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚183365510852096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 265536512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6505365504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26182 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1382 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8749331317248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8846504555520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26184 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚560 _step‚390 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8749331317248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52880084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192115075755008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172459996998656 _runtime‚5085.949259217 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8749564902912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183365492849152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90167143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚gÌÒ"( 217247577 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127080434 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172459979959296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚253 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217221393 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183365510852096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87160179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34279380 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289476608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192114824166400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222382436570112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90167141 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213535908074496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8722955139238 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8749564902912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8846239019008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 % _timestamp‚1.7884000223892474e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213535932014592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181183181107746 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222382147093504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90167141 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3516 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183365510852096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87160179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34279380 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 289476608 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192114824166400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222382436570112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90167141 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213535908074496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246009212 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8722955139238 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 530579456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251588608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 522164224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8749564902912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 517504849 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6505365504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233585664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8846239019008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6189583360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 530579456 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚253 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251588608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚307 % _timestamp‚1.7884000223892474e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32033 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 319781376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3516 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213535932014592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 108419072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181183181107746 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222382147093504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚253 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90167141 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127080327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751374064640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8848328594432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217265647 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90185322 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚718 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚222427354249728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192154027206656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285268992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8724987842738 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217291857 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181219911534770 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _step‚391 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8724758872678 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90185320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217265647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213579286984192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127106535 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183402653142016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87177956 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52890737 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183402635139072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213579310924288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172494906652672 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87177236 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚183402653142016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172494923692032 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 % _timestamp‚1.7884000232899933e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90185322 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8751140465664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90185320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52890739 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0´„®'´²éM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90185322 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8751140465664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚253 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90185320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52890739 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6189583360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251601920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8848067265536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6651354441 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127080327 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751374064640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233598976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚32 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8848328594432 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26210 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217265647 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90185322 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚718 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚222427354249728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192154027206656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 285268992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8724987842738 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233598976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217291857 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1354 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181219911534770 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚307 _step‚391 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6874205696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251601920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228970060 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8724758872678 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 517504849 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90185320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3516 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217265647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32033 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 522164224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213579286984192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127106535 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183402653142016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87177956 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52890737 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 319781376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183402635139072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213579310924288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172494906652672 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚307 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87177236 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚183402653142016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172494923692032 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7035944960 % _timestamp‚1.7884000232899933e+09‡.×Q 2³- _wandbruntime‚5086  _runtime‚5086$àAX´Ò 1loss/FBt4/assignment_loss‚0.22394427500513775 $ _timestamp‚1.788400023292156e+09 0loss/FRt2/assignment_loss‚4.8661225765943525 1loss/FBt3/assignment_loss‚0.17561309562370297 /loss/FRt2/detection_loss‚2.6117732614278792 .loss/FBt1/detection_loss‚3.186509234564645 epoch‚3 0loss/SRqqt2/assignment_loss‚2.24884579044122 0loss/FBt2/assignment_loss‚0.9243554877115717 .loss/FRt4/detection_loss‚2.087441996449516 /loss/FBt4/detection_loss‚0.8584728507778129 trainer/global_step‚4799 0loss/FRt4/assignment_loss‚2.9779331797645208 1loss/SRqqt1/assignment_loss‚2.092268295586109 _step‚392 1loss/FBt1/assignment_loss‚0.44478783008136297 %loss/total_loss‚54.10824576445032 _runtime‚ 5086.85220901 /loss/FRt3/assignment_loss‚4.438534520174327 2loss/SRqqt4/assignment_loss‚2.0578672999399714 0loss/SRqqt2/detection_loss‚2.000073378475813 .loss/FBt2/detection_loss‚2.980665990284511 /loss/FBt3/detection_loss‚2.9091886579990387 2loss/SRqqt3/assignment_loss‚2.1647137589752674 /loss/FRt1/detection_loss‚2.3503810660405593 /loss/FRt3/detection_loss‚2.6887631933940086 1loss/SRqqt1/detection_loss‚2.0303181327879427 1loss/SRqqt3/detection_loss‚2.1576660573482513 1loss/SRqqt4/detection_loss‚1.3327264031395316 0loss/FRt1/assignment_loss‚4.2992794318632646¥b]µ× 1loss/FBt1/assignment_loss‚0.44478783008136297 %loss/total_loss‚54.10824576445032 _runtime‚ 5086.85220901 /loss/FRt3/assignment_loss‚4.438534520174327 2loss/SRqqt4/assignment_loss‚2.0578672999399714 0loss/SRqqt2/detection_loss‚2.000073378475813 .loss/FBt2/detection_loss‚2.980665990284511 /loss/FBt3/detection_loss‚2.9091886579990387 2loss/SRqqt3/assignment_loss‚2.1647137589752674 /loss/FRt1/detection_loss‚2.3503810660405593 /loss/FRt3/detection_loss‚2.6887631933940086 1loss/SRqqt1/detection_loss‚2.0303181327879427 1loss/SRqqt3/detection_loss‚2.1576660573482513 1loss/SRqqt4/detection_loss‚1.3327264031395316 0loss/FRt1/assignment_loss‚4.2992794318632646 1loss/FBt4/assignment_loss‚0.22394427500513775 $ _timestamp‚1.788400023292156e+09 0loss/FRt2/assignment_loss‚4.8661225765943525 1loss/FBt3/assignment_loss‚0.17561309562370297 /loss/FRt2/detection_loss‚2.6117732614278792 .loss/FBt1/detection_loss‚3.186509234564645 epoch‚3 0loss/SRqqt2/assignment_loss‚2.24884579044122 0loss/FBt2/assignment_loss‚0.9243554877115717 .loss/FRt4/detection_loss‚2.087441996449516 /loss/FBt4/detection_loss‚0.8584728507778129 trainer/global_step‚4799 0loss/FRt4/assignment_loss‚2.9779331797645208 1loss/SRqqt1/assignment_loss‚2.092268295586109 _step‚392ˆûÊQ둶‚(:† ¤£ãÔÖ²èÒ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9818 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù–‘H·‚(:½ ¤£ãÔ”†ÚÕ disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 3125.5234375 disk.drbd1777.in‚0 cpu‚25.236326820002915 proc.cpu.threads‚17 network.recv‚ 1.781289e+06" disk.nvme0c0n1.in‚ 678.37890625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2846.33984375 network.sent‚ 1.1460982e+07) disk./.usagePercent‚70.82154410104572% disk.nvme0c0n1.out‚54963.99609375 disk.sda.in‚ 0.07421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚54963.99609375 disk.sda.out‚ 3125.5234375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.51611328125 disk.rbd0.out‚0# disk.nvme0n1.out‚54963.99609375 disk.rbd2.in‚0% disk./.usageGB‚1055.2921867370605 disk.rbd3.out‚0! disk.rbd2.out‚149.78369140625$ memory_percent‚98.92849922180176' proc.memory.availableMB‚ 175.5546875* proc.memory.percent‚17.372679710388184 disk.sda3.in‚ 0.07421875 disk.drbd1777.out‚0 disk.sda3.out‚ 3125.5234375 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 678.37890625 disk.dm-0.in‚ 678.378906255¬ï›¸‚(: ³£ãÔðÒ’Ó gpu.0.gpu‚39.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚86.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.271199999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05öÊM¹‚(: ³£ãÔºÜ¤Õ disk.drbd1777.in‚0$ memory_percent‚98.92854690551758) disk./.usagePercent‚70.82158301419092% disk.nvme0n1.out‚54967.5498046875 disk.sda.in‚ 0.07421875 disk.drbd1777.out‚0 cpu‚25.23578754091179 network.sent‚ 1.1568738e+07$ disk./.usageGB‚1055.292766571045! disk.rbd2.out‚149.78369140625 proc.cpu.threads‚17 disk.sda3.in‚ 0.07421875* proc.memory.percent‚17.372679710388184 network.recv‚ 1.78692e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 2.51611328125 disk.dm-1.out‚ 3126.12109375 disk.dm-0.in‚ 681.37890625" disk.dm-0.out‚54967.5498046875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 681.37890625 disk.nvme0n1.in‚ 681.37890625 disk.sda1.out‚0& proc.memory.availableMB‚ 175.546875# proc.memory.rssMB‚ 2846.33984375 disk.rbd2.in‚0 disk.sda3.out‚ 3126.12109375 disk.rbd0.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚54967.5498046875 disk.dm-1.in‚ 0.07421875 disk.rbd0.in‚0 disk.sda.out‚ 3126.12109375žo¢á‘º‚(:† £ãÔö€Ô gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×¥œÓN»‚(:à £ãÔææÖ disk.dm-1.out‚ 3126.7265625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 175.55859375 disk.sda2.in‚0( disk.nvme0c0n1.out‚55067.01025390625# disk.dm-0.out‚55067.01025390625* proc.memory.percent‚17.372679710388184 disk.sda2.out‚0 disk.sda3.in‚ 0.07421875 disk.dm-0.in‚ 681.7265625 network.sent‚ 1.1575085e+07 network.recv‚ 1.791755e+06 disk.rbd0.out‚0 disk.sda.in‚ 0.07421875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 681.7265625 disk.rbd0.in‚0& disk.nvme0n1.out‚55067.01025390625 disk.drbd1777.out‚0$ memory_percent‚98.92847537994385 disk.sda3.out‚ 3126.7265625 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 681.7265625 disk.dm-1.in‚ 0.07421875 disk.sda.out‚ 3126.7265625! disk.rbd2.out‚151.49853515625 disk.rbd1.out‚ 2.51611328125# proc.memory.rssMB‚ 2846.33984375 cpu‚25.24236548183704 proc.cpu.threads‚17) disk./.usagePercent‚70.82160272677106% disk./.usageGB‚1055.2930603027344 disk.rbd2.in‚0 disk.rbd3.out‚0æÒ@íš¼‚(: Ñ£ãÔ¼’èÒ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.237' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.647399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\¨ S½‚(:È Ñ£ãÔ¶Û¤×( disk.nvme0c0n1.out‚55068.99658203125 disk.rbd0.out‚0 network.sent‚ 1.1580629e+07) disk./.usagePercent‚70.82145321837106 disk.nvme0n1.in‚ 683.02734375 disk.sda1.in‚0 disk.sda3.out‚ 3127.33984375 disk.dm-0.in‚ 683.02734375 disk.rbd2.in‚0( proc.memory.availableMB‚ 174.80859375" proc.memory.rssMB‚ 2846.6640625& disk.nvme0n1.out‚55068.99658203125 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda3.in‚ 0.07421875" disk.nvme0c0n1.in‚ 683.02734375 disk.dm-1.in‚ 0.07421875* proc.memory.percent‚17.374658584594727 network.recv‚ 1.795931e+06 disk.rbd3.out‚0 disk.dm-1.out‚ 3127.33984375 disk.rbd0.in‚0 cpu‚25.255186109245596% disk./.usageGB‚1055.2908325195312 disk.sda2.out‚0 disk.rbd1.out‚ 2.51611328125# memory_percent‚98.9330530166626 proc.cpu.threads‚17# disk.dm-0.out‚55068.99658203125 disk.sda.out‚ 3127.33984375 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.sda.in‚ 0.07421875! disk.rbd2.out‚151.49853515625 disk.sda2.in‚0 disk.rbd1.in‚0“è.(š¾‚(: à£ãÔ«û×Ò gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.237' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.647399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vͤ«H¿‚(:½ à£ãÔÕ”¦Õ disk.nvme0n1.in‚ 683.02734375 disk.sda1.in‚0 disk.sda3.in‚ 0.07421875 disk.dm-1.in‚ 0.07421875$ memory_percent‚98.98793697357178! disk.rbd2.out‚151.63916015625 disk.rbd1.in‚0 disk.dm-1.out‚ 3127.96875) proc.memory.percent‚17.42985248565674 disk.sda.out‚ 3127.96875 disk.drbd1777.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚55168.2001953125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 3127.96875 disk.rbd1.out‚ 2.51611328125( proc.memory.availableMB‚ 165.81640625) disk./.usagePercent‚70.82145449840874 disk.dm-0.in‚ 683.02734375 disk.sda.in‚ 0.07421875 disk.sda1.out‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚55168.2001953125 disk.drbd1777.in‚0" disk.nvme0c0n1.in‚ 683.02734375 network.sent‚ 1.1586976e+07% disk./.usageGB‚1055.2908515930176 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2855.70703125 cpu‚25.268205658826112' disk.nvme0c0n1.out‚55168.2001953125 network.recv‚ 1.800562e+063²Ã2À- _wandbruntime‚5174  _runtime‚5174N[öÓF*ÁÀT VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188851288 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6189583360 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6505365504 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚91590446 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31808 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚225446954330624 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚183802269381430 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 8931086943744 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚32 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8833243343360 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20585 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚396 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚174995681980416 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚253 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20585 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31808 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1354 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚560 % _timestamp‚1.7884001106842406e+09 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20587 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚560 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 128455729 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚186052406443008 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 220025588 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚194885843536896 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6408446976 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 220025588 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8806587401014 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 220046175 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 319781376 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 _runtime‚ 5174.24453923 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚186052406443008 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚174995699019776 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚560 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193750528 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚91590446 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚91590444 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚34845410 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 530579456 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚194885843536896 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚194885649786368 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 295630336 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8806759212942 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 8931086943744 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚32 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8833243343360 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20585 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚396 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚174995681980416 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚253 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20585 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31808 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1354 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚560 % _timestamp‚1.7884001106842406e+09 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20587 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚560 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 128455729 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚186052406443008 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 220025588 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚194885843536896 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6408446976 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 220025588 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8806587401014 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 220046175 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 319781376 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 _runtime‚ 5174.24453923‰΀…î›Ã‚(: ï£ãÔï†ÐÕ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.830642700195312) gpu.0.memoryAllocatedBytes‚ 7585464320 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.237' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.647399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±!{ÄQÄ‚(:Æ ï£ãÔ—½šÖ$ memory_percent‚98.98688793182373( proc.memory.availableMB‚ 165.98828125 disk.rbd0.out‚0 cpu‚25.2803998597678) disk./.usagePercent‚70.82150416387039( disk.nvme0c0n1.out‚55169.44775390625 disk.sda2.in‚0 network.recv‚ 1.804433e+06$ disk./.usageGB‚1055.291591644287 disk.rbd1.in‚0 disk.dm-1.out‚ 3128.33984375# disk.dm-0.out‚55169.44775390625 disk.sda3.in‚ 0.07421875 disk.sda3.out‚ 3128.33984375" disk.nvme0c0n1.in‚ 683.74609375 disk.dm-0.in‚ 683.74609375 disk.sda1.in‚0 disk.drbd1777.out‚0! disk.rbd2.out‚151.63916015625# proc.memory.rssMB‚ 2855.70703125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 network.sent‚ 1.1592147e+07 disk.dm-1.in‚ 0.07421875 disk.nvme0n1.in‚ 683.74609375 disk.rbd2.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme0n1.out‚55169.44775390625 disk.sda.out‚ 3128.33984375) proc.memory.percent‚17.42985248565674 disk.rbd1.out‚ 2.51611328125 disk.sda.in‚ 0.07421875 disk.sda1.out‚0Q¡y2Å- _wandbruntime‚5175  _runtime‚5175=tˆ»Q)ÆËR \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚183828177171150 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚194912977050624 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1382 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚216544075458048 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚186079213198336 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚91605764 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 295630336 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚560 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6189583360 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚560 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚560 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 220049264 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8833588104704 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚216544099398144 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚175021057783808 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884001117065365e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32033 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚194912783300096 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚307 _runtime‚5175.266783698 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 220069851 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚91605766 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175747584 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 220049264 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚253 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 128443500 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚186079195195392 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚88585238 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚225475362046464 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6651354441 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6874205696 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚91605766 trainer/global_step‚4835 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 128464085 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20585 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚91605764 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚186079213198336 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 220069851 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚88584841 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1354 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 128464085 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3516 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚53733045 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145116672 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8807102347982 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20587 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6874205696 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20587 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7035944960 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8931286588416 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32033 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6505365504 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6408446976 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 128443500 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31808 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193750528 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚194912977050624 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚183827988319862 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚397 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8806930536054 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 522164224 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚395 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 530579456 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3516 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 517504849 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7035944960 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175747584 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8833763852288 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31808 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188851288 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 530579456 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚307 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8931431705088 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7035944960 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8833588104704 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 _step‚394 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚53733047 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34852191 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 319781376 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20585 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚194912783300096 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚225475531103232 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 108419072 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚560 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 220049264 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8833588104704 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚216544099398144 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚175021057783808 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884001117065365e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32033 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚194912783300096 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚307 _runtime‚5175.266783698 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 220069851 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚91605766 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175747584 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 220049264 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚253 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚253 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 128443500 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚186079195195392 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚88585238 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚225475362046464 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6651354441 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6505365504 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6874205696 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6408446976 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚91605766 trainer/global_step‚4835 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 128464085 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20585 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚32 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 530579456 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚91605764 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚186079213198336 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 220069851 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚88584841 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1354 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 128464085 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3516 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚53733045 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145116672 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8807102347982 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20587 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6874205696 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20587 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7035944960 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8931286588416 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32033 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6505365504 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6408446976 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 128443500 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31808 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193750528 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚194912977050624 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚183827988319862 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚397 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8806930536054 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 522164224 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚395 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 530579456 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3516 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 517504849 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7035944960 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175747584 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8833763852288 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31808 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188851288 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 530579456 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚307 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8931431705088 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7035944960 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6505365504 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8833588104704Šò·8a2È- _wandbruntime‚5189  _runtime‚5189ÿoBlÉ›¥ 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3223772644996643 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9825695753097534 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.27060890197753906 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.29697856307029724 7validation_average_jet_accuracy‚0.36044809222221375 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.012188208289444447 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909118413925171 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.352697491645813 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13137070834636688 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.32124605774879456 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08603888750076294 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_7_of_7‚0.5 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.015937330201268196 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06136046350002289 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2597222328186035 particle/accuracy_1_of_3‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04556650295853615 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05456290394067764 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3571545481681824 *jet/accuracy_1_of_3‚0.7392202615737915 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3743533492088318 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4736842215061188 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24629627168178558 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1114606112241745 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47967979311943054 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1226167157292366 particle/accuracy_5_of_6‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 *jet/accuracy_1_of_5‚0.8576101064682007 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19560393691062927 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2563376724720001 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.34166666865348816 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6483318209648132 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.05180180445313454 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.46436643600463867 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1625320464372635 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19697917997837067 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06362704932689667 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3560422360897064 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.41116034984588623 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9829309582710266 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9935152530670166 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37056878209114075 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4164726734161377 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30362698435783386 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Puri¨xœ{ty/1FRt*SRqqt*FBt/FBt_purity‚0.9899681806564331 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1431034356355667 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 & Purity/2FRt4SRqqt0FBt/FRt_purity‚0 *jet/accuracy_2_of_6‚0.3076923191547394 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3958333432674408 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03970111533999443 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.30740490555763245 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16070929169654846 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.053335223346948624 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9858402013778687 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.42351192235946655 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.06790123879909515 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3246321976184845 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957882165908813 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9797479510307312 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1378844976425171 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.11437681317329407 particle/accuracy_4_of_4‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015003779903054237 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.12654320895671844 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990444779396057 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.27879467606544495 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2388889491558075 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666269302368 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_1_of_4‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31891846656799316 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008354219025932252 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3464912474155426 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.003632762236520648 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 8 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.02777777798473835 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.033018868416547775 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.01458708941936493 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.04819023981690407 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3259129524230957 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_7‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.06948284804821014 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3008156716823578 _step‚395 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.052887462079524994 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06879245489835739 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.37726518511772156 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184398055076599 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3878968060016632 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3770981729030609 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.0966707095503807 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07240965962409973 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2667763829231262 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2777777910232544 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4088091552257538 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2686827778816223 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 jet/accuracy_4_of_7‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12518393993377686 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.29648715257644653 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2395833432674408 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9865092039108276 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,jet/accuracy_3_of_3‚0.012069589458405972 particle/accuracy_5_of_5‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_6‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 epoch‚3 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.27382928133010864 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14706239104270935 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984410405158997 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9749455451965332 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3548850417137146 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.040605224668979645 particle/accuracy_4_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 jet/accuracy_5_of_7‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3988780975341797 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06538160890340805 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889702796936035 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.37591320276260376 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9649305939674377 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0021027554757893085 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3199050724506378 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 )loss/val_total_loss‚52.91589505253814 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.27201157808303833 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2854938209056854 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11622365564107895 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_5‚0.3527841567993164 particle/accuracy_1_of_5‚1 +particle/specificity‚0.9474257230758667 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0444527268409729 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.26485076546669006 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9489942193031311 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3328065276145935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3115438222885132 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.015576476231217384 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3137730360031128 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0050977966748178005 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2885361909866333 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 _runtime‚5189.199978217 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3275221884250641 'particle/f_score‚0.2824557423591614 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.052887462079524994 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005538498051464558 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.40238627791404724 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_4_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6259685754776001 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.11436586827039719 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.15925925970077515 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.37726518511772156 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.27120041847229004 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 +particle/sensitivity‚0.2113078385591507 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.305555522441864 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3333333134651184 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.29741379618644714 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07153329253196716 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03704318404197693 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.37094205617904663 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3815721571445465 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09668923914432526 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28787878155708313 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2839505970478058 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03741338849067688 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.04508950188755989 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19568964838981628 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9832293391227722 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9901766777038574 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967023134231567 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.33786076307296753 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.10293444991111755 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4266025125980377 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3355957269668579 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.38286662101745605 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2727678716182709 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 jet/accuracy_3_of_7‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.37583127617836 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.34986773133277893 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8324859738349915 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2639433443546295 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3600192070007324 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.46062469482421875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6326071619987488 8 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.02777777798473835 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02738095447421074 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.25033673644065857 particle/accuracy_2_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.060674771666526794 9 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.051886796951293945 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.006666666828095913 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.27143537998199463 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3485020101070404 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.021473374217748642 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7884001256379333e+09 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.33000001311302185 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2990962862968445 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.023904267698526382 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08515369892120361 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.06321839988231659 -jet/accuracy_4_of_4‚0.0009920635493472219 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3826298117637634 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6060605645179749 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1974206268787384 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924680590629578 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1242658719420433 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_4‚0.806999683380127 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9885619878768921 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9905539751052856 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 trainer/global_step‚4835 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3555312156677246 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11257310211658478 & Purity/2FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.17446987330913544 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.05808081105351448 particle/accuracy_2_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20243756473064423 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2472972869873047 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3035714328289032 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07799245417118073 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09011843800544739 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.994235634803772 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.33196333050727844 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11895900219678879 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9600167274475098 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9908791184425354 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.33720073103904724 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.406242698431015 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_6_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.45529305934906006 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.39120492339134216 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19707505404949188 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4034721553325653 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.05734998732805252 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.04545454680919647 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2947540879249573 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08302252739667892 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3570590913295746 +jet/accuracy_2_of_2‚0.08249960094690323 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14898525178432465 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.008064515888690948 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38484883308410645 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.06249288469552994 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.826019287109375 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9038461446762085 +jet/accuracy_3_of_6‚0.09615384787321091 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32367220520973206 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.45854148268699646 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6212120652198792 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0014430014416575432 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984410405158997 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.40393149852752686 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0ÚéZVÊ ¥ ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11257310211658478 & Purity/2FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.17446987330913544 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.05808081105351448 particle/accuracy_2_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20243756473064423 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2472972869873047 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3035714328289032 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25ÆÛnñù :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07799245417118073 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09011843800544739 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.994235634803772 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.33196333050727844 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11895900219678879 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9600167274475098 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9908791184425354 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.33720073103904724 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.406242698431015 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_6_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.45529305934906006 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.39120492339134216 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19707505404949188 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4034721553325653 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.05734998732805252 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.04545454680919647 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2947540879249573 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08302252739667892 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3570590913295746 +jet/accuracy_2_of_2‚0.08249960094690323 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14898525178432465 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.008064515888690948 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38484883308410645 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.06249288469552994 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.826019287109375 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9038461446762085 +jet/accuracy_3_of_6‚0.09615384787321091 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32367220520973206 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.45854148268699646 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6212120652198792 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0014430014416575432 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984410405158997 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.40393149852752686 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3223772644996643 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9825695753097534 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.27060890197753906 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.29697856307029724 7validation_average_jet_accuracy‚0.36044809222221375 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.012188208289444447 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909118413925171 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.352697491645813 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13137070834636688 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.32124605774879456 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08603888750076294 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_7_of_7‚0.5 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.015937330201268196 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06136046350002289 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2597222328186035 particle/accuracy_1_of_3‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04556650295853615 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05456290394067764 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3571545481681824 *jet/accuracy_1_of_3‚0.7392202615737915 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3743533492088318 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4736842215061188 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24629627168178558 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1114606112241745 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.47967979311943054 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1226167157292366 particle/accuracy_5_of_6‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 *jet/accuracy_1_of_5‚0.8576101064682007 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19560393691062927 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2563376724720001 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.34166666865348816 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6483318209648132 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.05180180445313454 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.46436643600463867 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1625320464372635 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19697917997837067 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06362704932689667 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3560422360897064 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.41116034984588623 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9829309582710266 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.46875 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9935152530670166 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37056878209114075 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4164726734161377 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30362698435783386 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9899681806564331 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1431034356355667 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 & Purity/2FRt4SRqqt0FBt/FRt_purity‚0 *jet/accuracy_2_of_6‚0.3076923191547394 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3958333432674408 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03970111533999443 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.30740490555763245 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16070929169654846 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.053335223346948624 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9858402013778687 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.42351192235946655 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.06790123879909515 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3246321976184845 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9957882165908813 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9797479510307312 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1378844976425171 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.11437681317329407 particle/accuracy_4_of_4‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015003779903054237 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.12654320895671844 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990444779396057 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.27879467606544495 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2388889491558075 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666269302368 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_1_of_4‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31891846656799316 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008354219025932252 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3464912474155426 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.003632762236520648 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 8 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.02777777798473835 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.033018868416547775 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.01458708941936493 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.04819023981690407 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3259129524230957 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_7‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.06948284804821014 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3008156716823578 _step‚395 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.052887462079524994 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06879245489835739 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.37726518511772156 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184398055076599 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3878968060016632 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3770981729030609 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.0966707095503807 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07240965962409973 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2667763829231262 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2777777910232544 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4088091552257538 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2686827778816223 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 jet/accuracy_4_of_7‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12518393993377686 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.29648715257644653 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2395833432674408 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9865092039108276 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,jet/accuracy_3_of_3‚0.012069589458405972 particle/accuracy_5_of_5‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_2_of_6‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 epoch‚3 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.27382928133010864 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14706239104270935 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984410405158997 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9749455451965332 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3548850417137146 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.040605224668979645 particle/accuracy_4_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 jet/accuracy_5_of_7‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3988780975341797 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06538160890340805 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889702796936035 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.37591320276260376 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9649305939674377 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0021027554757893085 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3199050724506378 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 )loss/val_total_loss‚52.91589505253814 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.27201157808303833 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2854938209056854 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11622365564107895 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_5‚0.3527841567993164 particle/accuracy_1_of_5‚1 +particle/specificity‚0.9474257230758667 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0444527268409729 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.26485076546669006 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9489942193031311 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3328065276145935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3115438222885132 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.015576476231217384 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3137730360031128 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0050977966748178005 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2885361909866333 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 _runtime‚5189.199978217 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3275221884250641 'particle/f_score‚0.2824557423591614 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.052887462079524994 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005538498051464558 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.40238627791404724 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_4_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6259685754776001 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.11436586827039719 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.15925925970077515 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.37726518511772156 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.27120041847229004 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 +particle/sensitivity‚0.2113078385591507 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.305555522441864 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3333333134651184 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.29741379618644714 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07153329253196716 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03704318404197693 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.37094205617904663 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3815721571445465 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09668923914432526 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28787878155708313 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2839505970478058 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03741338849067688 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.04508950188755989 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19568964838981628 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9832293391227722 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9901766777038574 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967023134231567 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.33786076307296753 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.10293444991111755 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4266025125980377 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3355957269668579 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.38286662101745605 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2727678716182709 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 jet/accuracy_3_of_7‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.37583127617836 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.34986773133277893 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8324859738349915 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2639433443546295 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3600192070007324 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.46062469482421875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6326071619987488 8 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.02777777798473835 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02738095447421074 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.25033673644065857 particle/accuracy_2_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9946901202201843 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3472222089767456 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9 Purity/2FRt0SRqqt